Remote storage gateway management using gateway-initiated connections

ABSTRACT

A storage gateway serves as an interface between processes on a customer network and a service provider. The storage gateway is located on-premise with the customer processes. To customer processes, it appears that data is stored locally. However, the storage gateway interfaces with a remote storage service to store the data. For cached gateways, the primary data store is a remote data store, while frequently accessed data may be locally cached by the gateway. Reads may be satisfied from the local cache or from virtual data storage; writes are handled so as to appropriately update data blocks in the local cache and/or in virtual data storage. For shadowing gateways, the primary data store is the local data store; reads are passed through to a local data store, and writes are shadowed to virtual data storage as well as being sent to local data store.

This application is a continuation of U.S. application Ser. No.15/864,259, filed Jan. 8, 2018, which is a continuation of U.S.application Ser. No. 14/092,785, filed Nov. 11, 2013, now U.S. Pat. No.9,866,622, which is a divisional of U.S. application Ser. No.13/174,524, filed Jun. 30, 2011, now U.S. Pat. No. 8,601,134, which areherein incorporated by reference in their entirety.

BACKGROUND

Many companies and other organizations operate computer networks thatinterconnect numerous computing systems to support their operations,such as with the computing systems being co-located (e.g., as part of alocal network) or instead located in multiple distinct geographicallocations (e.g., connected via one or more private or publicintermediate networks). For example, data centers housing significantnumbers of interconnected computing systems have become commonplace,such as private data centers that are operated by and on behalf of asingle organization, and public data centers that are operated byentities as businesses to provide computing resources to customers. Somepublic data center operators provide network access, power, and secureinstallation facilities for hardware owned by various customers, whileother public data center operators provide “full service” facilitiesthat also include hardware resources made available for use by theircustomers. However, as the scale and scope of typical data centers hasincreased, the tasks of provisioning, administering, and managing thephysical computing resources have become increasingly complicated.

The advent of virtualization technologies for commodity hardware hasprovided benefits with respect to managing large-scale computingresources for many customers with diverse needs, allowing variouscomputing resources to be efficiently and securely shared by multiplecustomers. For example, virtualization technologies may allow a singlephysical computing machine to be shared among multiple users byproviding each user with one or more virtual machines hosted by thesingle physical computing machine, with each such virtual machine beinga software simulation acting as a distinct logical computing system thatprovides users with the illusion that they are the sole operators andadministrators of a given hardware computing resource, while alsoproviding application isolation and security among the various virtualmachines. Furthermore, some virtualization technologies are capable ofproviding virtual resources that span two or more physical resources,such as a single virtual machine with multiple virtual processors thatspans multiple distinct physical computing systems.

As another example, virtualization technologies may allow data storagehardware to be shared among multiple users by providing each user with avirtualized data store which may be distributed across multiple datastorage devices, with each such virtualized data store acting as adistinct logical data store that provides users with the illusion thatthey are the sole operators and administrators of the data storageresources.

Web Services

The conventional Web model allows clients to access Web resources (e.g.,applications, services, and data) via an HTTP client program, such as aWeb browser. A technology referred to as Web services has been developedto provide programmatic access to Web resources. Web services may beused to provide programmatic access to Web resources includingtechnology platforms (e.g., applications and services) and data (e.g.,product catalogs and other databases) hosted on Web-connected computerssuch as Web server systems via a Web service interface. Generallyspeaking, a Web service interface may be configured to provide astandard, cross-platform API (Application Programming Interface) forcommunication between a client requesting some service to be performedand the service provider. In some implementations, a Web serviceinterface may be configured to support the exchange of documents ormessages including information describing the service request andresponse to that request. Such documents, or messages, may be exchangedusing standardized Web protocols, such as the Hypertext TransferProtocol (HTTP), for example, and may be formatted in aplatform-independent data format, such as eXtensible Markup Language(XML), for example.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level block diagram of an example networkingenvironment that includes an example service provider and an exampleservice customer, according to at least some embodiments.

FIG. 2 illustrates an example architecture for and components of astorage gateway according to at least some embodiments.

FIG. 3 is a high-level block diagram of an example network environmentin which embodiments of a storage gateway may be implemented.

FIG. 4 is a block diagram of an example network environment thatincludes a storage gateway on site at a service customer network thatserves as an interface between the service customer network and astorage service on a service provider network, according to at leastsome embodiments.

FIG. 5 is a block diagram of an example service provider that provides astorage service and a hardware virtualization service to customers ofthe service provider, according to at least some embodiments.

FIG. 6 is a high-level block diagram that broadly illustrates thearchitecture of and data flow in an example network environment in whichan embodiment of a storage gateway is configured as a cached gateway.

FIG. 7 is a high-level block diagram that broadly illustrates thearchitecture of and data flow in an example network environment in whichan embodiment of a storage gateway is configured as a shadowing gateway.

FIG. 8 is a high-level block diagram that broadly illustratesbootstrapping a shadowing gateway in an example network environment,according to at least some embodiments.

FIG. 9 is a flowchart of a bootstrapping process for a shadowinggateway, according to at least some embodiments.

FIG. 10 is a flowchart of a shadowing gateway entering and recoveringfrom a pass-through mode, according to at least some embodiments.

FIG. 11 is a flowchart of a method for uploading, updating, and trackingblocks from a gateway to a remote data store, according to at least someembodiments.

FIG. 12 is a flowchart of an optimized bootstrapping process for ashadowing gateway, according to at least some embodiments.

FIG. 13 illustrates aspects of a storage gateway security model,according to at least some embodiments.

FIG. 14 is a flowchart that illustrates at least some aspects of agateway security model during activation, configuration, and operationof a storage gateway, according to at least some embodiments.

FIG. 15 is a high-level block diagram of an example networkingenvironment that illustrates the service customer and service providercomponents or entities that participate in a gateway activation process,according to at least some embodiments.

FIGS. 16A and 16B are process flow diagrams that illustrate interactionsamong the components illustrated in FIG. 15 during a gateway activationprocess, according to at least some embodiments.

FIGS. 17A and 17B are a flowchart of the activation process from theperspective of a storage gateway, according to at least someembodiments.

FIG. 18 is a high-level block diagram that illustrates example gatewaycontrol architecture that may be employed in at least some embodiments.

FIG. 19 is a flowchart of the method for remote gateway management usinga gateway-initiated connection and a long polling technique, accordingto at least some embodiments.

FIG. 20 is a flowchart of a method for a gateway control server tobroadcast a gateway request to its peer servers, according to someembodiments.

FIG. 21 is a flowchart of an alternative method for getting a gatewayrequest to the appropriate gateway control server, according to someembodiments.

FIG. 22 is a flowchart of a method for establishing, monitoring andmaintaining gateway-initiated connections, according to at least someembodiments.

FIG. 23A is a block diagram that broadly illustrates an architecture fora service provider network that includes a gateway proxy plane,according to at least some embodiments.

FIG. 23B illustrates a gateway control server messaging a gatewaythrough a gateway proxy plane, according to at least some embodiments.

FIG. 23C illustrates a gateway responding to a gateway control serverrequest through the gateway proxy plane, according to at least someembodiments.

FIG. 23D illustrates ping message exchange for a gateway proxy plane,according to at least some embodiments.

FIG. 24 illustrates a general architecture for and data I/O operationsof a cached gateway, according to at least some embodiments.

FIG. 25 illustrates a general architecture for and data I/O operationsof a shadowing gateway, according to at least some embodiments.

FIG. 26 is a flowchart of a method for writing to a write log on a blockdata store, according to at least some embodiments.

FIG. 27 is a flowchart of a method for satisfying a read request,according to at least some embodiments of a cached gateway.

FIG. 28 is a block diagram illustrating an example computer system thatmay be used in some embodiments.

While embodiments are described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that embodiments are not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit embodiments tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope as defined by the appended claims. The headings usedherein are for organizational purposes only and are not meant to be usedto limit the scope of the description or the claims. As used throughoutthis application, the word “may” is used in a permissive sense (i.e.,meaning having the potential to), rather than the mandatory sense (i.e.,meaning must). Similarly, the words “include,” “including,” and“includes” mean including, but not limited to.

DETAILED DESCRIPTION OF EMBODIMENTS

Various embodiments of methods, apparatus, and computer-accessiblestorage media for providing a local gateway to remote storage aredescribed. Embodiments of a storage gateway are described herein in thecontext of a service provider that provides, over an intermediatenetwork such as the Internet, a storage service to one or more customersof the service provider. The storage gateway may be implemented as avirtual or physical appliance that is installed on-premise at acustomer's data center and that acts as a gateway between the customer'sdata center and the storage service. The storage gateway may beconfigured as an interface to and local cache for a primary storageprovided remotely via the storage service and/or as an interface thatshadows primary storage implemented on the customer's network to remotestorage provided by the storage service. The storage gateway may presentstandard data access interfaces to the customer's applications at thefront-end of the appliance, convert the data accesses into storageservice requests at the back-end of the appliance, and transfer the dataover the network to the storage service according to the storage serviceinterface. In at least some embodiments, the storage service interfacemay be implemented as a Web service interface.

Embodiments of the storage gateway may provide an on-premise interfaceto virtually unlimited, flexible, scalable remote storage provided viathe storage service. The storage gateway may provide a cost-effective,flexible, and more easily scalable alternative to conventionalon-premise storage solutions. While the cost of storage devices may bedecreasing, the administrative and other hardware and software costs ofconventional on-premise storage solutions have remained relativelyconstant, or in some cases increased. Embodiments of the storage gatewaymay allow customers of a service provider to lower the total cost ofstorage ownership, passing at least some administrative and other coststo the service provider.

In at least some embodiments, the storage service may store thecustomer's data in the remote data store according to block storagetechnology. In at least some embodiments, the storage gateway may exposeblock storage protocols (e.g., iSCSI, GNBD (Global Network BlockDevice), etc.), file storage protocols (e.g., NFS (Network FileStorage), CIFS (Common Internet File System), etc.), and/or objectstorage protocols (e.g., REST (Representational State Transfer)) at thefront-end to the customer's applications. A block storage protocol suchas iSCSI enables direct access to the underlying data blocks of theremote data store.

Files written by an application to a remote data store via file storageprotocols such as NFS or CIFS exposed by the storage gateway may bestored to the remote data store according to block storage technology.Through an exposed file storage protocol such as NFS and CIFS, thestorage gateway presents the customer's data, stored in the remote datastore according to block storage technology, to the customer'sapplications as files before they are transmitted from the gateway overthe customer network to the customer's applications. The exposed blockstorage protocol, e.g. iSCSI, transfers the blocks to the customer'sapplications, thus requiring the application to handle interpretation ofthe data blocks into whatever format the application expects.

A block storage protocol such as iSCSI is a low-level block storageprotocol, and thus may enable a wider range of use cases than filestorage protocols such as NFS and CIFS. A block storage protocol mayenable support for applications that typically write to a block store,such as Microsoft® SharePoint® and Oracle® databases, and may also beconfigured to provide underlying storage for CIFS or NFS file servers.Thus, in at least some embodiments of the storage gateway, a blockstorage protocol such as iSCSI may be employed as the exposed interfaceto customer applications.

FIG. 1 is a high-level block diagram of an example networkingenvironment that includes an example service provider and an exampleservice customer, according to at least some embodiments. A storagegateway 84 may be installed, activated, and configured as a virtual orphysical appliance in the service customer local network or data center(e.g., client network 80) to provide one or more of several remote datastorage functionalities to customer process(es) 88 on the client network80. A customer process 88 may be any hardware, software, and/orcombination thereof that exists on the client network 80 and that canconnect to and communicate with the storage gateway 84 via the dataprotocol of the gateway 84's data ports (e.g., the iSCSI protocol). Thestorage gateway 84 may, for example, serve as an on-premise storagedevice and/or as an interface between the customer process(es) 88 on theclient network 80 and a storage service 64 provided by service provider60. Note that, in addition to a storage service 64, the service provider60 may also provide other services, including but not limited to ahardware virtualization service, to customers of the service provider60.

A customer of the service provider 60 may be referred to herein as aservice customer or simply customer, and may be any entity thatimplements a computer network or networks, coupled to an intermediatenetwork 50 such as the Internet, to provide networked computing servicesto one or more users on a local network or network, including one ormore services remotely provided by service provider 60. A servicecustomer may be a business enterprise, an educational entity, agovernment entity, or in general any entity that implements a computernetwork or networks that provide networked computing services to users.While FIG. 1 shows a single client network 80, there may be multipleclient networks 80. Each client network 80 may correspond to a differentservice customer, or two or more client networks 80 may correspond todifferent data centers or localities of the same service customer, forexample different regional offices of a business enterprise or differentcampuses of a school system. In at least some embodiments, each customerof the service provider 60 may have an account with the service provider60, and may be provided with security credentials (e.g., an account nameand/or identifier, password, etc.) via which one or more customerrepresentatives (e.g., a client network administrator) may log in tointerfaces (e.g., Web pages) to the service provider 60 to manage thecustomer's resources provided by one or more services, including but notlimited to a storage service, offered by the service provider 60.

Embodiments of storage gateway 84 may be implemented in hardware,software, or a combination thereof. In at least some embodiments,storage gateway 84 may be implemented as a virtual appliance that may,for example, execute within a virtual machine instantiated on a hostsystem. In at least some embodiments, storage gateway 84 may beimplemented as a virtual appliance that may be downloaded or otherwiseinstalled, activated, and configured on one or more computing devicessuch as server systems coupled to a local network infrastructure at aservice customer's data center (e.g., client network 80). Alternatively,storage gateway 84 may be implemented as a dedicated device or appliancethat may be coupled to a local network infrastructure at a servicecustomer's data center (e.g., client network 80); the dedicated deviceor appliance may include software and/or hardware that implements thefunctionality of the storage gateway 84. FIG. 26 illustrates an examplecomputer system on which embodiments of a storage gateway 84 may beimplemented. In at least some implementations, storage gateway 84communicates with the service provider 60 network via an intermediatenetwork 50 (e.g., the Internet) through firewall 82 technology. Notethat the service provider 60 network may also include front end 62technology (e.g., firewall technology, border router technology, loadbalancer technology, etc.) through which network traffic from and tointermediate network 50 passes.

At least some embodiments of the storage gateway 84 may be implementedaccording to a security model that provides data protection for thecustomer as well as protection against misuse and unauthorized use(e.g., pirating) of the gateway 84 by the customer or third parties.Communications between the storage gateway 84 and the storage service 64may be secured and encrypted. An activation process is described laterin this document in which a newly installed storage gateway 84 initiatesa connection with and is identified to the service provider 60 networkto obtain security credentials. In at least some embodiments, during theactivation process, the customer logs into the customer's account withthe service provider 60 and provides information to the service provider60 that is used in registering the gateway 84. However, the customerdoes not log in to the storage gateway 84, and therefore the customer'ssecurity credentials and other account information are not exposed onthe gateway 84. This may minimize the security risk for the customer.

In at least some embodiments, an aspect of the security model is thatthe storage gateway 84 only accepts externally-initiated connections toone or more data ports (e.g., iSCSI ports) exposed to the customerprocess(es) 88 on the client network 80. The storage gateway initiatesall other connections to external processes; external processes cannotinitiate any other connections to the gateway. For example, in at leastsome embodiments, the storage gateway 84 initiates gateway managementand other connections to the service provider 60; the service provider60 does not initiate connections to the gateway 84. As another example,a client network 80's network administrator process 90 cannot directlyconnect to the storage gateway 84 to configure and manage the gateway84. Instead, configuration and management of the storage gateway 84 bythe network administrator process 90 may be performed through theservice provider 60, for example via console process 68 on the serviceprovider 60 network. Thus, in at least some embodiments, a user, networkmanager, or process (e.g., network administrator process 90 or customerprocess(es) 88) on the client network 80 cannot directly “log in” to thestorage gateway 84, nor can a user, manager, or process on the serviceprovider 60 network (e.g., console process 68 and storage service 64) oron some other external network initiate a connection to the storagegateway 84. This helps protect the security credentials and otheroperational information on the storage gateway 84 from beingintentionally or unintentionally compromised by persons or processes onthe client network 80 or by external persons or processes.

Embodiments of the storage gateway 84 may be installed, activated, andconfigured for use with a storage service 64 to provide one or more ofseveral data store 66 functionalities. For example, a storage gateway 84may be installed, activated, configured, and employed with a storageservice 64 to serve as:

-   -   A file system gateway. In this configuration, the storage        gateway serves as a NAS storage interface (e.g., using CIFS or        NFS protocols) to the storage service 64. The remote data store        66 may be presented to the customer by the gateway 84 as an        object store (e.g., REST), while the data store 66 is        implemented according to block storage technology. In this        configuration, the remote data store 66 may be presented to the        customer as a virtualized file system to which the customer can        write files and from which the customer can read files.    -   A cloud volume gateway. In this configuration, the storage        gateway 84 serves as an interface to volume(s) implemented on        remote data store 66 via the storage service 64. The remote data        store 66 may be implemented using block storage technology. The        gateway 84 provides local network access points, with the        volume(s) on remote data store 66 (which may also be referred to        as a cloud volume) serving as backend storage that provides        flexible and essentially unlimited primary storage capacity. In        this configuration, the remote data store 66 may be presented to        the customer as a cloud volume system from which the customer        can locally mount volumes for reading and writing data.    -   A shadowing gateway. In this configuration, the storage gateway        84 acts as a “bump in the wire” between a customer's        applications (e.g., customer process(es) 88) and the customer's        local data store 86 to provide shadowing of the customer's write        data (e.g., iSCSI writes) to remote data store 66 via the        storage service 84. The remote data store 66 may be implemented        using block storage technology. In this configuration, the        storage gateway 84 may serve as a shadowing appliance that        shadows the customer's local data store to snapshot(s) on the        remote data store 66. This shadowing may be performed        transparently from the perspective of users on the local        network. When necessary or desired, the customer may request or        access snapshot(s) of the customer's data on the remote data        store 66, for example to restore, recover, or copy portions or        all of the customer's data from the snapshot(s) to a local store        86.

Note that the file system gateway and the cloud volume gateway aresimilar in that both serve as gateways to a remote data store, and bothmay locally cache data, e.g. frequently and/or recently used data. Inboth the file system gateway and the cloud volume gateway, data readsfrom customer processes may be serviced from the local cache, ifpossible, or from the remote data store if not. In contrast, in theshadowing gateway, data reads are passed through the gateway to thecustomer's local data store. For the purposes of this document, the filesystem gateway and cloud volume gateway may collectively be referred toas a cached gateway to distinguish these implementations from theshadowing gateway.

Example Storage Gateway Appliance Architecture

FIG. 2 illustrates an example architecture for and components of astorage gateway according to at least some embodiments. Note that someof the components illustrated in FIG. 2 may not be used, or may be usedor implemented differently, in shadowing gateway implementations whencompared to cached gateway implementations.

Block driver 10 interfaces a customer process 88 with the storagegateway 84. generally, block driver 10 allows a customer process 88 tointeract with the storage gateway 84 (e.g., via read/write requests).Since the storage gateway 84 is on-site with the customer process 88,from the perspective of the process 88 it appears that data is storedlocally. However, the storage gateway 84 interfaces with storage service64 to store the data to a remote data store 66 provided by the storageservice 64. For cached gateways, the primary data store is remote datastore 66, while frequently accessed data may be locally cached by thegateway 84. Reads may be satisfied from the local cache or from virtualdata storage 66; writes are handled so as to appropriately update datablocks in the local cache and/or in virtual data storage 66. Forshadowing gateways, the primary data store is local data store 86; readsare passed through to local data store 86, and writes are shadowed tovirtual data storage 66 as well as being sent to local data store 86.

Block driver 10 intercepts read/write requests from the customer process88 and passes the requests to the storage controller 12. In at leastsome embodiments, block driver 10 may provide a block storage protocol(e.g., iSCSI or GMBD) as an interface to the customer process 88. Insome embodiments, instead of or as an alternative to a block storageprotocol interface, block driver 10 may provide a file storage protocolinterface (e.g., NFS or CIF S) and may use file system semantics as aninterface to the storage controller 12. Note that, while FIG. 2 showsone block driver 10, there may be more than one block driver.

Storage controller 12 acts as a mediator between block driver 10 andstorage via a cache manager 14. Responsibilities of storage controller12 may include forwarding read and write requests from block driver 10to storage and callbacks to block driver 10 when storage responds withdata. Block driver 10 may also maintain statistics such as the number ofrequests in progress.

In at least some embodiments, storage controller 12 on one storagegateway 84 may communicate with a cache manager 14 on another storagegateway 84. In at least some embodiments, each storage gateway 84 maysend heartbeat messages for discovery and detecting failures. Aconsistent hashing may be used to identify the storage gateway 84 thatis responsible for a given object, and the request to get data may beforwarded to the cache manager 14 on the target storage gateway 84. Thecache manager 14 may respond by invoking a callback provided by storagecontroller 12.

In cached gateway embodiments, cache manager 14 may manage a local cache28 that, for example, provides storage for frequently accessed data.Local cache 28 may be implemented on internal volatile and/ornon-volatile memory of storage gateway 84, or alternatively may beimplemented at least in part on an external local data store 86 providedby the customer. In at least some embodiments, the local cache 28represents data stored in the virtualized data storage 66; writes from acustomer process 88 may not directly affect the local cache 28.

In at least some embodiments employing multiple gateways 84, adistributed local cache may be used, and consistent hashing on keys maybe used to identify the cache responsible for holding a given key. In atleast some embodiments, locality-aware request distribution may be usedto reduce communication between the gateways 84, which may requireadditional load balancing.

All write requests to a given volume in the remote data store 66 may goto a particular gateway 84 node. Since all write requests for a volumeare forwarded to a particular gateway 84 node, network partitioning maynot be an issue.

Staging

In at least some embodiments, the cache manager 14 may include or mayinterface with a staging 16 component. Staging 16 may include or mayhave access to a write log 18. In at least some embodiments, a datastructure may be built over the write log 18 and used as a metadatastore 26. The metadata store 26 may allow quick access to all writes toa particular block. The metadata store 26 may, for example, be used inapplying mutations to different segments within the block. When writedata is received from the customer process 88, the data is appended tothe write log 18. Metadata for the write data relative to a block, e.g.offset and length, may be stored to the metadata store 26. In at leastsome embodiments, write log 18 may be implemented as a one-dimensionaldata buffer implemented as either a linear or a circular queue. In atleast some embodiments, metadata store 26 may be a key/value store, forexample implemented as a Berkeley Database. Other implementations ofboth the write log 18 and the metadata store 26 may be used in someembodiments.

In cached gateway implementations, when a read is performed, theoriginal block may be obtained from the local cache 28 or from theremote data store 66, and any pending mutations indicated by the writelog 18 may be applied before returning the data to the respectivecustomer process 88.

In some embodiments, if a gateway 84 fails (e.g. crashes), in-memorywrite data may be lost unless the data has already been written to thelocal data store 86. In some embodiments, if there are multiple gateways84 at the customer site, another gateway 84 may take responsibility ofkeys owned by the crashed gateway 84, restore writes from a snapshot onlocal data store 86 if there are any, and start accepting requestsdirected to the respective volume. In some embodiments, a write log 18and/or metadata store 26 may be replicated over two or more gateways 84to provide redundancy and better durability. In case of failure of thegateway 84, one of the other gateways 84 may take over the failedgateway's write log 18 and metadata store 26. However, in at least someembodiments, the metadata store 26 may be maintained only on the ownergateway 84. In these embodiments, in case of failure of the gateway 84,one of the other gateways 84 may take over and parse the primary writelog 18 to rebuild the metadata store 26.

In cached gateway implementations, block fetcher 22 fetches requiredsegments of blocks from remote data store 66 via storage service 64. Inat least some embodiments, block fetcher 22 may employ a lazy fetchingtechnique to fetch complete blocks for caching. For both cached gatewaysand shadowing gateways, block store 24 pushes data from staging 16 toremote data store 66 via storage service 64. In at least someembodiments, block store 24 may employ a lazy pushing technique to pushthe blocks.

In at least some embodiments, during read operations for cachedgateways, block driver 10 sends the read request including a volume ID,start offset and length to storage controller 12. In at least someembodiments, storage controller 12 may translate the volume ID andoffset to to an object key. Storage controller 12 may pass the readrequest information to cache controller 14, which may attempt to satisfythe read request from an appropriate local cache 28. If the data are notpresent in the local cache 28, the request is forwarded to block fetcher22, which fetches the data from the appropriate volume on remote datastore 66 via storage service 64. Once the data is obtained, local cache28 is updated, mutations from write log 18 are applied, and a readresponse is returned to customer process 88. In at least someembodiments, if multiple blocks are requested, multiple read responsesmay be returned each indicating a relative offset for a respectiveblock. In at least some embodiments, if sequential reads are detected,sequential blocks may be prefetched.

In at least some embodiments, during write operations, block driver 10sends the write request including a volume ID and the write data to thestorage controller 12 that is responsible for the volume. The write datais written to the write log 18, and metadata store 26 is updated toinclude a reference to the mutated data in buffer pool 20.

Buffer Pool

In at least some embodiments, a buffer pool 20 resides between storagecontroller 12 and local data store 86. Buffer pool 20 may perform one ormore of, but not limited to, the following tasks. Note that some tasksmay apply only to cached gateways:

-   -   Cache data for the logical offsets for write log 18 and local        cache 28 from their physical locations on local data storage        device(s).    -   Maintaining locks on buffers during read and write operations.    -   Applying an eviction technique, e.g. a least recently used (LRU)        based eviction technique, on the physical storage for local        cache 28. Note that this is not required for shadowing gateways.    -   For reads in cached gateways, if the requested data is not found        in local cache 28, buffer pool 20 may communicate with block        fetcher 22 to fetch the block from remote data store 66.        Alternatively, in some embodiments, block fetcher 22 may        communicate directly with storage service 64 to fetch blocks.

In at least some embodiments, buffer pool 20 may employ a database, forexample a Berkeley database (BDB), as its metadata store 26. Table 1,shown below, shows information that may be stored in a metadata store26, according to at least some embodiments. Note that the entries inTable 1 are not intended to be limiting according to content orarrangement.

TABLE 1 Example metadata store information Physical Last Disk/OffsetType Name Offset Used <sdg/xxxxx> F (Free) N/A N/A N/A <sdg/xxxxx> B(Bad) N/A N/A N/A <sdg/xxxxx> W (Write log) N/A write log offset <time><sdg/xxxxx> S (Snapshot) snapshot ID offset in volume <time> <sdg/xxxxx>C (Chunk) chunk ID offset in volume <time>

In at least some embodiments, the physical disk offset is at a setboundary, for example at a 4 MB boundary. In at least some embodiments,this includes boundaries for data in both the volumes and in the writelog 18. In at least some embodiments, the writes for a specific volumemay be sequential writes, and thus fragmentation on disk may not need tobe considered. Note that a “chunk” may correspond to a block, or to oneor more blocks.

Note that the metadata store 26 may include both S (snapshot) and C(chunk) entries, and these need to be kept up-to-date with the schemevia which the storage controller 12 attempts to access blocks. Forexample, a block may be referred the first time using a snapshot ID, butevery time after that using the chunk ID. This may be preserved in themetadata store 26. Upon a Snapshot Complete, storage controller 12 mayrefer to the blocks from the snapshot using the snapshot ID; hence, theC (chunk) entries in metadata store 26 may be converted intocorresponding S (snapshot) entries.

Cached Gateway Operations

In at least some embodiments, when a read request is received, the writelog 18 entry or entries for the block are looked up in the metadatastore 26. If the read request can be satisfied using the write log 18entry or entries, then all required entries are looked up in themetadata store 26, read into buffers, flattened, and the required piecesare returned. If the read request cannot be satisfied only using thewrite log 18 entry or entries, the offset for the cache data block(e.g., a 4 MB block) is calculated from the offset in the read request.The location of the block is looked up in the metadata store 26. If theblock is in local cache 28, the block is read from the local cache 28,and if not it is fetched from remote data store 66. The required writelog 18 entries are fetched as described above, flattened with the block,and and the required pieces are returned. If the block is fetched fromremote data store 66, the block is cached to local cache 28 and recordedin the metadata store 26. The last access time for the block in thelocal cache 28 is also updated.

In at least some embodiments, when a write request is received, themutations are recorded at the next write log 18 offset and the metadata,i.e. offset and length, is recorded in the metadata store 26.

In at least some embodiments, when a block upload completes, the latestversion of the block (with the applied mutations) is added to the localcache 28 and recorded in the metadata store 26. If a previous version ofthe block is present in local cache 28, this block is marked as free inmetadata store 26.

In at least some embodiments, when a snapshot completes, the metadatastore 26 may need to be reorganized as described above. That is, theblock entries belonging to the snapshot may be converted into thecorresponding snapshot entries on the remote data store 66.

Shadowing Gateway Operations

In at least some embodiments, read requests are passed through to localdata store 86.

In at least some embodiments, when a write request is received, thewrite data is recorded at the next write log 18 offset and theappropriate metadata for the write is recorded in the metadata store 26.The write request is also passed to the local data store 86.

In at least some embodiments, to upload a block to remote data store 66,an upload process calls buffer pool 20 to read the write log 18. Thebuffer pool 20 uses metadata store 26 to perform the translation fromthe logical write log 18 offset to the physical offset, and the data isthen read into memory buffers. The buffers are then presented to theupload process. The upload process uploads the blocks to the remote datastore 66 and releases the blocks to the buffer pool 20.

Write Log Purges

In at least some embodiments, if the write log 18 needs to be purged,buffer pool 20 obtains a write log offset for a volume for which thewrite log 18 can be purged. In at least some embodiments, the write logoffset may be determined from metadata store 26, for example byperforming a walk over the database which checks offsets for each entry.To purge the write log 18, the existing write log entries correspondingto the purgeable part of the log may be marked as free entries.

Example Implementations

FIG. 3 is a high-level block diagram of an example network environmentin which embodiments of a storage gateway may be implemented. A serviceprovider 110 on an intermediate network 100 (e.g., the Internet) mayprovide one or more service customer networks (e.g., client network(s)150), also coupled to intermediate network 100, access to a remote datastore 116 via a storage service 112. Each client network 150 maycorrespond to a different service customer, or two or more clientnetworks 150 may correspond to different data centers or localities ofthe same service customer, for example different regional offices of abusiness enterprise or different campuses of a school system. A servicecustomer may be a business enterprise, an educational entity, agovernment entity, a private entity, or in general any entity thatimplements a computer network or networks, coupled to an intermediatenetwork 100 such as the Internet, to provide networked computingservices to one or more users. In some embodiments, storage service 112may provide an interface, for example a Web service interface, via whicheach service customer's client network(s) 150 may access functionalityprovided by the storage service 112.

Customer processes 154A and 154B represent physical and/or virtualmachines or systems connected to a client network 150 of a servicecustomer. As an example of a function provided by storage service 112, auser, via a customer process 154, may create and mount data volumes inremote data store 116 via storage service 112. From the perspective ofusers on a client network 150, the data volumes provided by storageservice 112 may appear as if they are local storage; hence, such a datavolume may be referred to as a virtual data volume 158. A virtual datavolume 158 actually maps to one or more physical storage devices orstorage systems on which remote data store 116 is instantiated; however,this mapping is handled by the storage service 112, and is thustransparent from the perspective of the users on the client network 150.A user of a customer process 154 may simply see a volume mounted on thedesktop or in a device listing. The user of a customer process 154 maycreate data, modify data, delete data, and in generally perform anydata-related function on virtual data volume 158, just as if the volume158 was implemented on a locally attached storage device.

FIG. 4 is a block diagram of an example network environment thatincludes a storage gateway 252 on site at a service customer's clientnetwork 250 that serves as an interface between client network 250 andstorage service 212, according to at least some embodiments. In at leastsome embodiments, storage gateway 252 may be a file and/or block storageappliance that is installed on-site at a service customer's data center.

Storage gateway 252 may, for example, be installed, activated, andconfigured to serve as a file system gateway, as a cloud volume gateway,collectively referred to as cached gateways, or as a shadowing gateway.A file system gateway serves as a NAS storage interface (e.g., usingCIFS or NFS protocols) to the storage service 212. The remote data store216 may be presented to the customer as an object store (e.g., REST),while actually implemented as block storage. A cloud volume gatewayserves as an interface to virtualized volume storage provided by thestorage service 212. The volume storage may be implemented as blockstorage. The gateway 252 provides local network access points, with theremote data store 216 (which may also be referred to as a cloud volume)serving as backend storage that provides flexible and essentiallyunlimited primary storage capacity. A shadowing gateway acts as a “bumpin the wire” between a customer's applications and the customer's localdata store to provide shadowing of the customer's write data (e.g.,iSCSI writes) to remote storage provided by the storage service 212. Theremote data store 216 may be implemented as block storage.

In cached gateway implementations, storage gateway 252 may store a localcache of frequently accessed data on a local data store 254, whilesecurely encrypting and accelerating data movement back to serviceprovider 210. Similarly, shadowing gateway implementations may securelyencrypt and accelerate the movement of write data to service provider210. This accelerated data movement, as compared to a standard Internetconnection, may, for example, be achieved using one or more of datadeduplication, compression, parallelization, and TCP window scalingtechniques. Storage gateway 252 may significantly reduce the cost,utilization, maintenance, and provisioning headaches that are typicallyassociated with managing on-site storage arrays as primary storage orbackup storage. Storage gateway 252 may accomplish this by replacing the100 s of terabytes to petabytes of data a customer may otherwise storein-house on expensive hardware, e.g. NAS or SAN hardware, with acost-effective appliance. With the storage gateway 252, customers maybenefit from the low access latencies of on-site storage (provided bythe local cache maintained by the gateway 252 in cached gatewayimplementations) while leveraging the durable, available, and scalabledistributed storage infrastructure provided by the service provider 210.

Embodiments of the storage gateway 252 may work seamlessly withcustomers' on-site applications. In at least some embodiments, customersmay configure the storage gateway 252 to support SAN (iSCSI), NAS (NFS,Microsoft® CIFS), or Object (REST) storage. In at least someembodiments, an iSCSI interface provided by the storage gateway 252 mayenable integration with on-site block storage applications such asMicrosoft® SharePoint® and Oracle® databases. In at least someembodiments, customers may utilize NFS and CIFS interfaces provided bythe storage gateway 252 to consolidate file storage across environmentsincluding, but not limited to, Windows, Linux, and UNIX environments. Inat least some embodiments, the storage gateway 252 may also beconfigured to support REST-based requests.

In at least some embodiments, storage gateway 252 may be implemented asa virtual device or appliance that may be downloaded or otherwiseinstalled, activated, and configured on one or more computing devicessuch as server systems coupled to the client network 250 infrastructureat a customer data center. Alternatively, storage gateway 252 may beimplemented as a dedicated device or appliance that may be coupled tothe client network 250 infrastructure; the dedicated device or appliancemay include software and/or hardware on which functionality of thegateway may be implemented.

In at least some implementations, storage gateway 252 communicates withthe service provider 210 network via an intermediate network 200 (e.g.,the Internet). The coupling of storage gateway 252 to intermediatenetwork 200 may generally be via a high-bandwidth connection provided bythe service customer's client network 250, as large amounts of data maybe transferred across intermediate network 200 between storage service212 and storage gateway 252. For example, at peak times, the connectionmay need to support the transfer of data at rates of 100 megabits/second(100 Mbit/s) or higher. However, in at least some embodiments,techniques such as a data deduplication technique may be employed toreduce bandwidth usage when uploading data from storage gateway 252 tostorage service 212, and thus more of the connection's bandwidth may beavailable for other applications. Example data deduplication techniquesthat may be employed in at least some embodiments are described in U.S.patent application Ser. No. 12/981,393, titled “RECEIVER-SIDE DATADEDUPLICATION IN DATA SYSTEMS,” which is hereby incorporated byreference in its entirety, and in U.S. patent application Ser. No.12/981,397, titled “REDUCED BANDWIDTH DATA UPLOADING IN DATA SYSTEMS,”which is hereby incorporated by reference in its entirety.

In at least some embodiments, bandwidth on a connection between clientnetwork 250 and service provider 210 over intermediate network 200 maybe allocated to storage gateway 252, and to other customer applications,for example via a network administrator process 260 at client network250. Storage gateway 252 may continuously or nearly continuously uploadmutated (new or changed) data to storage service 212, for exampleaccording to a data deduplication technique. However, the mutation rateof data at client network 250 may vary over time; for example, duringthe day, the customer process write throughput may be higher, while atnight the write throughput may be lower. Thus, at busy times when themutation rate is high, storage gateway 252 may fall behind in uploadingthe mutated data if the bandwidth allocated to the storage gateway 252is not high enough to keep up; storage gateway 252 may then catch up atless busy times when the mutation rate is not as high. In at least someembodiments, if the storage gateway 252 falls behind more than aspecified threshold, the storage gateway 252 may request the allocationof additional bandwidth. In at least some embodiments, the storagegateway 252 may raise an alarm to demand more bandwidth, if necessary.

While FIG. 4 shows a direct connection between storage gateway 252 andstorage service 212, note that the connection between storage gateway252 and storage service 212 may go through local network 256.

In at least some embodiments of a storage gateway 252, rather thanretrieving data from remote data store 216 on demand, large blocks orchunks of data, even entire volumes of data, may be locally cached to alocal data store 254. Storage gateway 252 may include or may have accessto physical data storage and/or memory (local data store 254) on which alocal cache of data, for example frequently-accessed data or criticaldata, may be maintained. Local data store 254 may be volatile ornon-volatile storage or memory, or a combination thereof. Maintaining alocal cache of frequently accessed data may generally improve dataaccess times for customer processes 258, since many or most dataaccesses can be serviced from the local cache, rather than retrievingthe data from remote data store 216. However, remote data store 216 mayserve as the primary data store for the service customer's clientnetwork 250; thus, storage gateway 252 may communicate with storageservice 212 via an intermediate network 200 to periodically,aperiodically, or continuously upload new or modified data from thelocal cache to remote data store 216, and to download requested datafrom remote data store 216 when necessary.

In FIG. 4, storage (218A, 218B, 218C, . . . ) of remote data store 216illustrates that the remote data store 216 may be implemented on oracross several storage devices or systems connected to a local network214 of service provider 210. Thus, a service customer's data may bespread across two or more physical storage devices or systems on the“back end.” The back end storage devices may be, but are notnecessarily, multi-tenant devices that are shared with other customers.However, as noted in reference to FIG. 3, from the perspective of theusers and processes on client network 250, the client's data may bepresented as virtual volumes or files.

In at least some embodiments, a service provider as described inreference to FIGS. 3 and 4 may also provide hardware virtualizationtechnologies and possibly other virtualization technologies tocustomers. A service provider 200 may provide a range of virtualizedcomputing technology and virtualized storage technology, including blockstorage technology that provides block storage capabilities (i.e., ablock-based storage system) to customers. Virtual computing environmentsor systems, implemented according to the hardware virtualizationtechnology provided by the service provider 200, may be supported by theblock storage technology. The block storage technology may provide avirtualized storage system that, for example, is able to interact withvirtual computing systems through standardized storage calls that renderthe block-level storage functionally agnostic to the structural andfunctional details of the volumes that it supports and to the operatingsystems executing on the virtual computing systems (or other systems) towhich it provides storage availability.

Embodiments of a storage gateway 252 may integrate with on-site customerapplications and the virtualized computing and storage technologyprovided by service provider 200, providing customers with access toelastic “cloud-based” computing and storage resources. For example,customers using the storage gateway 252 for SAN storage may createconsistent, point-in-time block-based snapshots of their data. Thesesnapshots may then be processed by hardware virtualization technologyapplications or instances (see, e.g., virtual computing system(s) 264 inFIG. 5) requiring the high I/O and low latency data access that ablock-based storage system provides. As another example, customers mayconfigure the storage gateway 252 for NAS storage via NFS or CIFS fileprotocols, and may create point-in-time snapshots of their file dataaccessible from hardware virtualization technology instances.

In some embodiments, objects written using a REST-based interfaceprovided by storage gateway 252 may be accessed directly fromvirtualized storage technology provided by the service provider via HTTPor other protocols, or may be distributed using integrated contentdelivery technology provided by the service provider. In someembodiments, customers may also utilize highly scalable, distributedinfrastructure provided by the virtualized storage technology forparallelized processing of these objects on hardware virtualizationtechnology instances.

FIG. 5 is a block diagram of an example service provider that provides astorage service and a hardware virtualization service to customers ofthe service provider, according to at least some embodiments. A servicecustomer's client network 250 may include one or more storage gateways252 that serve as interfaces between client network 250 and storageservice 212 of service provider 210, for example as described inreference to FIG. 4. Service client(s) may represent any administrator,user, or process that may access one of the services provided by serviceprovider 210.

Hardware virtualization technology may enable multiple operating systemsto run concurrently on a host computer 292, i.e. as virtual machines(VMs) 296 on the host 292. The VMs 296 may, for example, be rented orleased to the customers of the service provider 210. A hypervisor, orvirtual machine monitor (VMM) 294, on a host 292 presents the VMs 296 onthe host 292 with a virtual platform and monitors the execution of theVMs 296. Each VM 296 may be provided with one or more IP addresses; theVMM 294 on a host 292 may be aware of the IP addresses of the VMs 296 onthe host. A local network of service provider 210 may be configured toroute packets from the VMs 296 to Internet destinations (e.g., toservice client(s) 262 on client network 250), and from Internet sources(e.g., service client(s) 262) to the VMs 296.

Service provider 210 may provide a service customer's client network250, coupled to intermediate network 200 via local network 256, theability to implement virtual computing systems 264 via a hardwarevirtualization service 290 coupled to intermediate network 200 and tothe local network of service provider 210. In some embodiments, hardwarevirtualization service 290 may provide an interface, for example a Webservice interface, via which a service client 262 may accessfunctionality provided by the hardware virtualization service 290. Atthe service provider 210, each virtual computing system 264 mayrepresent a virtual machine (VM) 296 on a host 292 system that isleased, rented, or otherwise provided to a service customer.

From an instance of a virtual computing system 264, a user may accessthe functionality of storage service 212 as previously described. Thus,embodiments of a virtualized system as illustrated in FIG. 5 may allow aclient to create local instances of virtual computing systems 264implemented on VMs 296 provided by the service provider 210, and toaccess data from and store data to a remote data store 216 implementedby the service provider 210, from the local instances of the virtualcomputing systems 264.

As previously described, one or more storage gateways 252 may beinstantiated at the client network 250. At least one of the gateways 252may be a cached gateway implementation that locally caches at least somedata, for example frequently accessed or critical data. The storagegateway(s) 252 may communicate with storage service 212 via one or morehigh-bandwidth communications channels, for example to upload new ormodified data from the local cache so that the primary store of data(the remote data store 216) is maintained in cached gatewayimplementations, or to upload new or modified data (write data) to asnapshot of a local primary data store on remote data store 216 inshadowing gateway implementations.

Cached Gateway Implementations

FIG. 6 is a high-level block diagram that broadly illustrates thearchitecture of and data flow in an example network environment in whichan embodiment of a storage gateway is configured as a file systemgateway or as a cloud volume gateway, which may be collectively referredto as cached gateways. In at least some embodiments, storage gateway 252may be a file and/or block storage appliance that is installed on-siteat a service customer's data center. In FIG. 6, storage gateway 252 may,for example, be installed, activated, and configured to serve as a filesystem gateway or as a cloud volume gateway. A file system gatewayserves as a NAS storage interface (e.g., using CIFS or NFS protocols) tothe storage service 212. The remote data store 216 may be presented tothe customer as an object store (e.g., REST), while implemented as blockstorage. A cloud volume gateway serves as an interface to virtualizedvolume storage provided by the storage service 212. The virtualizedvolume storage may be implemented as block storage. The gateway 252provides local network access points, with the remote data store 216(which may also be referred to as a cloud volume) serving as backendstorage that provides flexible and essentially unlimited primary storagecapacity.

Once storage gateway 252 is installed, activated, and configured, anetwork administrator process 260 of client network 250 may, forexample, create new data volumes 270 or mount existing data volumes 270on remote data store 216 via storage service 212. Create volume requestsand other service requests may be made to the service 212 via serviceprovider front end 280. The front end 280 may also manage connectionsand communications to and from storage gateway 252. The front end 280may include one or more of, but is not limited to, firewalls, borderrouters, load balancers, gateway servers, gateway proxies, consoleprocesses, and in general any networking device and/or process that maybe necessary to expose the storage service 212 to client network(s) 250and to interface the storage service 212 to storage gateway(s) 252.

In at least some embodiments, storage gateway 252 initiates allconnections to the service provider 210 via service provider front end280; the service provider 210 does not initiate connections to thegateway 252. In addition, the network administrator process 260 does notinitiate connections directly to the gateway 252; access by the networkadministrator process 260 to the gateway 252, for example to configureand manage the gateway 252, is through the service provider 210 viaservice provider front end 280.

Storage gateway 252 exposes one or more data ports (e.g., iSCSI ports)to the customer process(es) 258 on the client network 250. A customerprocess 258 may be any hardware, software, and/or combination thereofthat exists on the client network 250 and that can connect to andcommunicate with the storage gateway 252 via the data protocol of thegateway 252's data ports (e.g., the iSCSI protocol). A customer process258 may be, for example, a storage application such as Microsoft®SharePoint® and Oracle® databases, a server (e.g., an SQL server, aMicrosoft® Exchange® server, etc.), a database application (e.g., an SQLdatabase application, and Oracle® database application), a Microsoft®Exchange® application, or any other application or process executing onone or more devices on the client network 250 that is operable tocommunicate with the storage gateway 252 data port(s). Note that acustomer process, as used herein, encompasses any software process thatmay be executing on one or more devices in the client network 250;however, the underlying hardware on which the process executes may beinvolved in or perform the connections and communications to the storagegateway 252 data port(s) on behalf of the process.

A mounted volume 270 may be presented to the customer process(es) 258 bystorage gateway 252. Customer process(es) 258 may then perform readsfrom and writes to the volume 270 via the data ports exposed by thestorage gateway 252, for example according to iSCSI protocol. Storagegateway 252 handles all read and write requests to volume 270. While thevolume(s) 270 on remote data store 216 serves as the primary data store,storage gateway 252 may also store a local cache of frequently accesseddata on a local data store 254. Local data store 254 may be implementedon storage hardware internal to the storage gateway 252, on storagehardware external to the storage gateway 252 provided by the servicecustomer, or on a combination thereof.

For reads, storage gateway 252 may first check the local cache to see ifa given read can be satisfied from the cache. If the read cannot besatisfied from the local cache, then storage gateway 252 may request thedata from storage service 212, which gets the requested data (or a blockor chunk of data that includes the requested data) from remote datastore 216 and returns the requested data to the storage gateway 252.Storage gateway 252 may store the block or chunk of data received fromstorage service 212 to the local cache.

For writes, storage gateway 252 may write the new or updated data to thelocal cache. In at least some embodiments, the write data may beappended to a block-based write log implemented in the local cache.Storage gateway 252 may include a sender-side data upload process (notshown) that communicates with a receiver-side data upload process (notshown) at service provider 210 to periodically, aperiodically, orcontinuously upload new or modified data in the local cache to theprimary data store 216. The uploading of write data from the write logmay be performed asynchronously to the processing of the read and writeoperations from the initiating processes to the local data store 254. Inat least some embodiments, this upload process may employ one or more ofdata deduplication, compression, parallelization, and TCP window scalingtechniques. Example data deduplication techniques that may be employedin at least some embodiments as illustrated in FIG. 6 are described inU.S. patent application Ser. Nos. 12/981,393 and 12/981,397, which werepreviously incorporated by reference in their entireties.

The local cache may be limited in size, while the remote data store 216may provide essentially unlimited storage space. Thus, storage gateway252 may remove, replace, or overwrite older and/or relatively inactivedata blocks in the local cache with newer and/or active data blocks.

Shadowing Gateway Implementations

FIG. 7 is a high-level block diagram that broadly illustrates thearchitecture of and data flow in an example network environment in whichan embodiment of a storage gateway is configured as a shadowing gateway.In FIG. 7, storage gateway 252 may be installed, activated, andconfigured to serve as a shadowing gateway that acts as a “bump in thewire” between a customer's applications and the customer's local datastore to provide shadowing of the customer's write data (e.g., iSCSIwrites) to remote storage provided by the storage service 212. Theremote data store 216 may be implemented as block storage.

In the embodiment illustrated in FIG. 7, local data store 254 serves asthe primary data store for the customer process(es) 258 on clientnetwork 250, in contrast to the cached gateway implementation in FIG. 6where remote data store 216 serves as the primary data store. Oncestorage gateway 252 is installed, activated, and configured as ashadowing gateway, the storage gateway 252 exposes one or more dataports (e.g., iSCSI ports) to the customer process(es) 258 on the clientnetwork 250. The customer process(es) 258 on client network 250 may thenread from and write to the local data store 254 via the storage gateway252 data port(s). A customer process 258 may be any hardware, software,and/or combination thereof that exists on the client network 250 andthat can connect to and communicate with the storage gateway 252 via thedata protocol of the gateway 252's data ports (e.g., the iSCSIprotocol). A customer process 258 may be, for example, a storageapplication such as Microsoft® SharePoint® and Oracle® databases, aserver (e.g., an SQL server, a Microsoft® Exchange® server, etc.), adatabase application (e.g., an SQL database application, and Oracle®database application), a Microsoft® Exchange® application, or any otherapplication or process executing on one or more devices on the clientnetwork 250 that is operable to communicate with the storage gateway 252data port(s). Note that a customer process, as used herein, encompassesany software process that may be executing on one or more devices in theclient network 250; however, the underlying hardware on which thecustomer process executes may be involved in or perform the connectionsand communications to the storage gateway 252 data port(s) on behalf ofthe process.

The read and write requests may be received by the gateway 252 dataport(s). For reads, the requests may be passed directly to the localdata store 254 without further interference or processing by gateway252, and the requested data may be passed directly from local data store254 to customer process 258. Write requests directed to the local datastore 254 are also passed to the local data store 254 by storage gateway252. However, in addition to passing the write requests to the localdata store 254, the storage gateway 252 may shadow the new or updateddata indicated by the write requests to the remote data store 216 viathe storage service 212.

In at least some embodiments, to shadow new or updated data to theremote data store 216, storage gateway 252 may locally store or bufferthe write data to be uploaded to the to the remote data store 216, forexample in a first-in-first-out (FIFO) write log. In at least someembodiments, the write log may be implemented in a block storage format,with the write log comprising one or more blocks (e.g., 4 MB blocks).Write data received in the write requests may be appended to the writelog. The write data from two or more write requests may be written tothe same block in the write log. Metadata for the write data relative toa block, e.g. offset in the write log block and length, as well as anoffset in the target data store, may be stored to a metadata store.

Storage gateway 252 may include a sender-side data upload process (notshown) that communicates with a receiver-side data upload process (notshown) at service provider 210 to periodically, aperiodically, orcontinuously upload the locally stored write data from the write log tothe shadowed data volume at remote data store 216. The uploading ofwrite data from the write log may be performed asynchronously to theprocessing of the read and write operations from the initiatingprocesses to the local data store 254. The upload process may upload thewrite data from the write log in blocks. Once a write log block has beensuccessfully uploaded, the corresponding block may be marked as free inthe write log.

In at least some embodiments, the upload process may employ one or moreof data deduplication, compression, parallelization, and TCP windowscaling techniques. Example data deduplication techniques that may beemployed in at least some embodiments as illustrated in FIG. 7 aredescribed in U.S. patent application Ser. Nos. 12/981,393 and12/981,397, which were previously incorporated by reference in theirentireties.

Note that a service provider front end 280 may manage connections tostorage gateway 252. In at least some embodiments, storage gateway 252initiates connections to the service provider 210 via front end 280; theservice provider 210 does not initiate connections to the gateway 252.The front end 280 may include one or more of, but is not limited to,firewalls, border routers, load balancers, gateway servers, gatewayproxies, console processes, and in general any networking device and/orprocess that may be necessary to expose the storage service 212 toclient network(s) 250 and to interface the storage service 212 tostorage gateway(s) 252.

In at least some embodiments, storage gateway 252 initiates allconnections to the service provider 210 via service provider front end280; the service provider 210 does not initiate connections to thegateway 252. In addition, the network administrator process 260 does notinitiate connections directly to the gateway 252; access by the networkadministrator process 260 to the gateway 252, for example to configureand manage the gateway 252, is through the service provider 210 viaservice provider front end 280.

As a shadowing gateway, the shadowing operations provided by the storagegateway 252 may be effectively transparent to from the perspective ofusers on the client network 250. The customer process(es) 258 performreads and writes to the data port(s) (e.g., iSCSI port(s)) exposed bythe storage gateway 252 on the client network 250. From the customerprocess 258 perspective, the storage gateway 252 may appear as any otherdata target (e.g., iSCSI target). Read requests from the customerprocess(es) 258 received on the data port(s) are passed on to the localdata store 254 that serves as the primary data store. Write requestsfrom the customer process(es) 258 received on the data port(s) arepassed on to the local data store 254 and shadowed to the remote datastore 216. The shadowing operations of the gateway 252 may be performedin the background without significantly affecting performance of theprimary data store or of the client network 250.

An example use case for the “bump in the wire” shadowing gatewayconfiguration illustrated in FIG. 7 is for disaster recovery. Storagegateway 252 sends updates of data from client network 250 to storageservice 212, which stores the data in a shadow volume or volumes, alsoreferred to as a snapshot 272. The data may be stored in the snapshot272 in a block storage format. The data are also stored to a local datastore 254. If something happens that results in the corruption or lossof a portion or all of a locally stored volume, the corrupted or lostdata may be recovered from a snapshot 272 of the volume stored in datastore 216. Storage provider 210 may provide an interface via which acustomer network administrator (e.g., via network administrator process260) may request the recovery of a snapshot 272 of a portion or all of alocally stored volume from a shadowed volume on remote data store 216.In at least some embodiments, at least a portion of the write logmaintained by storage gateway 252 may be uploaded to the remote datastore 216 prior to recovering a snapshot 272 of the data to ensure thatthe shadowed volume from which data is to be recovered is as up-to-dateas possible. Note that, in some cases, at least some data may berecovered directly from the write log maintained by storage gateway 252.

Customer Process-Gateway Communications

As previously described, a customer administrator, via networkadministrator process 260, may communicate with storage gateway 252(e.g., a shadowing gateway) via the service provider 280 front end, forexample to configure the gateway 252. In at least some embodiments, oneor more customer processes 258 may also be configured to communicatewith the storage gateway 252 via the service provider 280 front end tomake requests of the gateway 252. For example, a customer process 258may be an SQL server that is configured to communicate with storagegateway 252 via the service provider 280 front end.

Shadowing Gateway Bootstrapping Techniques

As illustrated in FIG. 7, once storage gateway 252 is installed,activated, and configured as a shadowing gateway, the storage gateway252 exposes one or more data ports (e.g., iSCSI ports) to the customerprocess(es) 258 on the client network 250. The customer process(es) 258on client network 250 may then read from and write to the local datastore 254 via the storage gateway 252 data port(s). The read and writerequests are passed to the local data store 254, and the write dataindicated by the write requests are shadowed to the remote data store216 so that snapshot(s) 272 of the local data store may be updated.

However, when a shadowing gateway comes online in a customer's network,either when initially installed, activated and configured or after beingoffline for some reason, there may be data in the local data store 254that is not in the snapshot(s) 272 on the remote data store 216. Thus,at least some embodiments may provide a bootstrapping process forshadowing gateways during which at least some data from the local datastore 254 may be uploaded to the remote data store 216 so that thesnapshot(s) can be populated and/or updated to accurately reflect thedata that is currently on the local data store 254.

FIG. 8 is a high-level block diagram that broadly illustratesbootstrapping a shadowing gateway in an example network environment,according to at least some embodiments. When storage gateway 252 comesonline as a shadowing gateway on the client network 250, the gateway 252may determine that there is data in the local data store 254 that needsto be uploaded to the remote data store 216 to make the snapshot 272consistent with the local data store 254. An upload process of thegateway 252 may then begin to upload blocks of data from the local datastore 254 to the remote data store 216 at service provider 210. Thestorage gateway 252 may also expose its data ports to customerprocess(es) 258, begin accepting and processing read requests and writerequests directed to the local data store 254, begin caching the newwrite data indicated by the write requests to the write log, and beginuploading the write data from the write log to the remote data store216. The upload of data from the local data store 254 may thus beperformed in the background while the storage gateway 252 is performingits shadowing function on the client network 250. When the upload ofdata from the local data store 254 is complete, the storage gateway 252continues performing its shadowing function.

FIG. 9 is a flowchart of a bootstrapping process for a shadowinggateway, according to at least some embodiments. As indicated at 300, ashadowing gateway comes online on a customer's network. For example, anew instance of a storage gateway may be installed, activated, andconfigured as a shadowing gateway on the network. As another example, anexisting instance of a shadowing gateway may come back online afterbeing offline for some reason; while the gateway was offline, customerprocess(es) may have communicated directly to the local data store toread and write data. As another example, a shadowing gateway may haveentered a pass-through mode during which shadowing operations aretemporarily suspended for some reason, for example due to the write logbecoming full, and may be exiting the pass-through mode and resumingshadowing operations.

As indicated at 302, the shadowing gateway may begin uploadingpre-existing data from the local data store to the remote data store, ifnecessary. For example, if this is a new shadowing gateway and the localdata store is already populated, the existing data in the local datastore needs to be uploaded to the remote data store so that a consistentsnapshot can be generated. As another example, if an existing shadowinggateway comes back online or resumes shadowing operations upon exitingpass-through mode, new data may have been written to the local datastore, and thus the snapshot on the remote data store needs to be madeconsistent with the data currently on the local data store.

As indicated at 304, the shadowing gateway may begin accepting reads andwrites from the customer processes via the gateway data port(s) exposedon the customer's network. As indicated at 306, the shadowing gatewaymay begin caching write data from the writes to a write log, and beginuploading write data from the write log to the remote data store asindicated at 308.

The upload of data from the local data store begun at 302 may beperformed in the background while the shadowing gateway accepts read andwrite requests and performs its shadowing function on the customer'snetwork. When the upload of data from the local data store is complete,the shadowing gateway continues performing its shadowing function.

Note that the order of the elements in FIG. 9 may be different. Forexample, element 302 may be performed after any one of elements 304through 308. In other words, the shadowing gateway may begin acceptingreads and writes and performing its shadowing function prior tobeginning to upload the pre-existing data from the local data store.

FIG. 10 is a flowchart of a shadowing gateway entering and recoveringfrom a pass-through mode, according to at least some embodiments. Asindicated at 320, a shadowing gateway may enter a pass-through mode bysuspending its shadowing function (i.e., stop caching and uploadingwrite data) while continuing to accept and service reads and writesdirected to the local data store from the customer processes on thecustomer's network. The gateway may enter pass-through mode upondetecting some condition that may cause the shadowing function to fail.As an example, the shadowing gateway may enter the pass-through modeupon detecting that the write log is full and cannot be successfullyuploaded. The gateway may alert the local network administrator of thedetected condition; the administrator may then address the problemindicated by the alert. For example, the administrator may allocate morememory to the write log, and/or allocate more bandwidth to the gatewayupload process. The administrator may then inform the gateway that theproblem has been addressed.

When the shadowing gateway determines that the pass-through mode can beexited, for example by receiving an indication that a detected problemthat caused the pass-through mode has been addressed, the gateway mayrestart shadowing (i.e., start caching and uploading write data), asindicated at 322.

Upon exiting pass-through mode, there may be data in the local datastore that has not been uploaded to the remote data store. Since thegateway continues to receive and process write requests duringpass-through mode, new data may have been written to the local datastore. Thus, the shadowing gateway may perform a bootstrap asillustrated in FIGS. 8 and 9 to upload at least some data from the localdata store to the remote data store to recover from the pass-throughmode, as indicated at 324.

In at least some embodiments, an optimized bootstrapping process forshadowing gateways may be employed to reduce the amount of data that isuploaded from the local data store to the remote data store. Theoptimized bootstrapping process may detect blocks of data that havealready been uploaded to the remote data store, and thus avoid uploadingblocks that have already been uploaded. The optimized bootstrappingprocess may leverage tracking data that is generated and maintained fora storage gateway process during general uploading of data from agateway to the remote data store.

FIG. 11 is a flowchart of a method for uploading, updating, and trackingblocks from a gateway to a remote data store, according to at least someembodiments. During normal gateway operations, the gateway uploads writedata to the remote data store at the service provider, specifically tothe storage service, as indicated at 360. The storage service receivesthe write data and gets the respective block(s) (e.g., 4 MB blocks) fromthe remote data store as indicated at 342. The storage service thenmodifies the respective block(s) according to the write data and uploadsthe modified block(s) back to the remote data store with a new versionname, as indicated at 344. For each modified block, a token indicatingthe modified block is sent back to the storage gateway, as indicated at346. The storage gateway keeps track of these tokens; every time a blockis modified, the reference block that is being modified needs to be sentto the storage service.

As indicated at 348, the storage gateway may periodically oraperiodically update a token manifest at the service provider and purgeat least a portion of the locally tracked tokens. The storage gatewaymay have to track a large number of tokens. In at least someembodiments, a manifest may be provided on the remote data store thatmay relieve the storage gateway of the burden of having to locally tracka large number of tokens. The storage gateway may periodically oraperiodically call the storage service to update the manifest withtoken(s) that the gateway has received, and may purge the respectivelocally stored tokens.

In at least some embodiments, the optimized bootstrapping process mayleverage the manifest to determine what blocks have and have not beenuploaded by making a call to check hashes of each of the blocks in themanifest to determine which blocks indicated by the manifest matchblocks on the local data store versus which blocks indicated by themanifest do not match blocks on the local data store and thus need to beuploaded. In other words, the manifest is used to detect which blocks onthe local data store are dirty blocks, and which are not. Thus, theoptimized bootstrapping process attempts to determine, via the manifest,which blocks have already been uploaded so that the already-uploadedblocks are not uploaded again, and only dirty blocks are uploaded. In atleast some embodiments, for the blocks that the optimized bootstrappingprocess determines do need to be uploaded (the dirty blocks), a datadeduplication technique may be applied when uploading these blocks toreduce the amount of data that is actually uploaded from the dirtyblocks.

FIG. 12 is a flowchart of an optimized bootstrapping process for ashadowing gateway, according to at least some embodiments. Abootstrapping process may be started for a shadowing gateway, forexample when the gateway exits pass-through mode. As indicated at 360, ablock is obtained from the local data store. As indicated at 362, themanifest, which may be stored on the remote data store, may be checkedto determine if the current block is a dirty block that needs to beuploaded. At 364, if the current block is dirty according to themanifest, at least a portion of the block may be uploaded to the remotedata store according to a data deduplication technique, as indicated at366. The method then proceeds to 368. At 364, if the current block isnot dirty according to the manifest, the method proceeds directly to368. At 368, if more blocks are to be processed, the method returns toelement 360 to process a next block. Otherwise, the bootstrappingprocess is done.

Storage Gateway Security Model

Embodiments of the storage gateway may be implemented according to asecurity model that provides data protection for the customer as well asprotection against misuse and unauthorized use (e.g., pirating) of thegateway by the customer or third parties. FIG. 13 illustrates aspects ofa storage gateway security model, according to at least someembodiments.

In at least some embodiments, an aspect of the security model is that astorage gateway 84 is delivered and initially installed on a clientnetwork 80 without security credentials or other identifying informationfor the gateway 84 to use in communications with the service provider60. An activation process may be employed via which a storage gateway 84on a customer network can register with the service provider 60. In atleast some embodiments of the activation process, the storage gateway 84may initiate a connection (e.g., an SSL (Secure Socket Layer)/TCPconnection) with and identify itself to the service provider 60 as acorrect gateway for a respective customer account to obtain thenecessary security credentials. During the activation process, theservice customer specifies a name for the gateway 84. In at least someembodiments, the service customer logs into the customer's account withthe service provider 60 and provides information to the service provider60, including but not limited to the gateway name, that is used inregistering the gateway 84. However, the service customer does not login to the storage gateway 84, and therefore the service customer'ssecurity credentials and other account information are not exposed onthe gateway 84. This may minimize the security risk for the servicecustomer. This gateway name, along with other metadata related to thegateway 84 and to the service customer, may be stored by the serviceprovider 60 and used in tracking and identifying the respective gateway84. Note that a service customer may have one or more gateways 84installed and activated on a client network 80, with each having aunique identifying name and other metadata. FIGS. 15 through 17B,further described below in the section titled Storage gateway activationprocess, illustrate an activation process that may be employed in atleast some embodiments. In the activation process, the gateway 84 mayinitiate a connection to the service provider 60 and provide metadataabout the gateway 84 platform, along with a public key, to the serviceprovider 60. The service provider 60 may then provide a temporary,unique activation key to the gateway 84 that is used in the activationprocess. In addition, a service customer may be required to log in tothe customer's account via a service provider console process toactivate the gateway 84; thus, the gateway 84 can be matched with theaccount of the service customer that attempts to activate the gateway84. The security credentials and other metadata (e.g., thecustomer-supplied gateway name) obtained by the storage gateway 84 viathe activation process may then be used by the storage gateway 84 incommunications with various processes of the service provider 60 networkto identify the gateway 84 to the service provider 84 processes.

In at least some embodiments, another aspect of the security model, asillustrated in FIG. 13, is that the storage gateway 84 only acceptsexternally-initiated connections to one or more data ports (e.g., iSCSIports) exposed to the customer process(es) 88 on the client network 80.The storage gateway does not accept other externally initiatedconnections, and initiates all necessary connections to externalprocesses. For example, in at least some embodiments, the storagegateway 84 initiates at least one secure connection 92 (e.g., an SSL(Secure Socket Layer)/TCP connection) to the service provider 60; theservice provider 60, however, cannot initiate connections to the gateway84. An example method for remote gateway management usinggateway-initiated connections and a long polling technique that may beused in at least some embodiments is illustrated in FIGS. 18 through 20.

In addition, as illustrated in FIG. 13, in at least some embodiments,the service customer (e.g., network administrator process 90) does notdirectly connect to the storage gateway 84 to configure and manage thegateway 84; instead, configuration and operation requests for thestorage gateway 84 are made through the service provider 60, whichpasses the requests to the gateway 84 via the secure communicationschannel 92 initiated by the gateway 84. For example, as illustrated inFIGS. 18 through 21, configuration and operation requests for a gateway84 may be performed by or via a network administrator process 90 througha console process on the service provider 60 network. In at least someembodiments, the console process forwards a received configurationrequest or operation request directed to the customer's gateway 84 to agateway control plane that maintains gateway-initiated connections 92.The gateway control plane locates a current connection to the gateway 84that is the target of the request, for example a connection maintainedon a particular gateway control server, and the request is forwarded tothe gateway 84 via the connection.

Thus, in at least some embodiments, a user, network administrator, orprocess of the customer cannot directly initiate connections to or “login” to the storage gateway 84, nor can external persons or processessuch as an operator or process on the service provider 60 networkinitiate a connection to the storage gateway 84. This, along with otheraspects of the gateway security model, may help to protect the securitycredentials and other operational information on the storage gateway 84from being intentionally or unintentionally compromised by externalpersons or processes.

In another aspect of the security model, all communications between thestorage gateway and the storage service during activation and operationof the gateway may be secured and encrypted. As noted above, an aspectof the security model is that communications between the storage gatewayand the storage service are performed over gateway-initiated secureconnections (e.g., SSL/TCP connections). An encryption technique, forexample public/private key encryption, may be used in communicationsover the gateway-initiated secure connections.

FIG. 14 is a flowchart that illustrates at least some aspects of agateway security model during activation, configuration, and operationof a storage gateway, according to at least some embodiments. Asillustrated at 400, a storage gateway may be instantiated on a customernetwork. For example, to instantiate the storage gateway, the storagegateway may be installed as a virtual or physical appliance on theservice customer's local network or data center, typically behind afirewall. For example, in at least some embodiments, the storage gatewaymay be implemented as a virtual appliance that may be downloaded to orotherwise installed on one or more computing devices such as serversystems on the service customer's local network. Alternatively, thestorage gateway may be implemented as a dedicated device or appliancethat may be coupled to the service customer's local network; thededicated device or appliance may include software and/or hardware thatimplements the functionality of the storage gateway. As illustrated at402, the instantiated storage gateway initiates an activation processwith the service provider and the customer to identify the gateway andto obtain gateway security credentials. In at least some embodiments,the security credentials include a certificate signed with agateway-provided public key. An example activation process is describedbelow in reference to FIGS. 15 through 17B. Note that the activationprocess may be initiated by the gateway when the gateway is initiallyinstalled on the customer network, and may also be initiated at othertimes, for example when powering on after the gateway device has beenpowered down for upgrade, maintenance, or for some other reason. Asindicated at 404 of FIG. 14, the storage gateway establishes a secureconnection to the service provider. An example method for agateway-initiated connection that uses a long polling technique that maybe used in at least some embodiments is illustrated in FIGS. 18 through21. As indicated at 406 of FIG. 14, the customer configures and operatesthe storage gateway through a service provider console process. Anexample method for remote gateway management using gateway-initiatedconnections and a long polling technique that may be used in at leastsome embodiments is illustrated in FIGS. 18 through 21. As illustratedat 408 of FIG. 14, the storage gateway communicates with the serviceprovider, for example to communicate with a storage service process,using the gateway security credentials and possibly other metadataobtained during the activation process to identify the gateway to theservice provider.

Storage Gateway Activation Process

Embodiments of a storage gateway may, for example, serve as anon-premise storage device and as an interface between a servicecustomer's network and a storage service provided by a service provider.In at least some embodiments, the storage gateway may be implemented asa virtual device or appliance that may be downloaded or otherwiseinstalled on one or more computing devices such as server systemscoupled to a local network infrastructure of the customer at a customerdata center. Alternatively, the storage gateway may be implemented as adedicated device or appliance that may be coupled to a local networkinfrastructure of the customer. The dedicated device or appliance mayinclude software and/or hardware that implements the functionality ofthe gateway.

In at least some embodiments, in order to use a storage gateway afterthe gateway is installed, the gateway must be activated with the serviceprovider. This section describes a method via which identification,authentication, and authorization of a storage gateway may be performedduring bootstrapping, or activation, of the storage gateway. In thegateway activation method, the storage gateway is identified andassociated with the customer's service provider account. However, thecustomer's credentials are not exposed to the storage gateway during theactivation process. In at least some embodiments, the customer logs intothe customer's account with the service provider and providesinformation to the service provider, including but not limited to agateway name, that is used in registering the gateway 84. However, thecustomer does not log in to the storage gateway, and therefore thecustomer's security credentials and other account information are notexposed on the gateway. This may minimize the security risk for thecustomer. In at least some embodiments, the service provider accountthat is used by the customer in the activation process may be the sameaccount that the customer used to manage other resources that areprovided to the customer by the service provider, including but notlimited to other storage resources provided by a storage service andvirtualized hardware resources provided by a hardware virtualizationservice, as illustrated in FIG. 5.

FIG. 15 is a high-level block diagram of an example networkingenvironment that illustrates the service customer and service providercomponents or entities that participate in a gateway activation process,according to at least some embodiments. These participants may include,but are not limited to, a storage gateway 84, a network administratorprocess 90, a console process 68, and gateway control 70 a (e.g., of aplurality of gateway control process 70 a/70 b). A storage gateway 84may be installed as a virtual or physical appliance on a servicecustomers local network or data center (e.g., client network 80),typically behind a firewall. For example, a storage gateway 84 may be avirtual appliance that, for example, executes within a virtual machine,and may be downloaded and instantiated on a server device on clientnetwork 80. A console process 68 a on the service provider 60 networkmay be accessible by or via a network administrator process 90, forexample from a device on client network 80 or from a device external toclient network 80, to sign on to the customer's account. For example,the console process 68 may provide a web interface or some otherinterface via which a network administrator, via network administratorprocess 90, may sign on to the respective service customer's account toview and manage the account and resources provided by the serviceprovider 60. A gateway control 70 process or plane of the serviceprovider 60 network may perform tracking and management functions forone or more storage gateway(s) 84 installed at one or more customers ofthe service provider 60. Gateway control 70 and console process 68 amay, for example, be implemented on one or more server computer deviceson service provider 60 network. In at least some embodiments, gatewaycontrol 70 may be implemented as a control plane that includes two ormore gateway control servers to provide load balancing and highavailability.

FIGS. 16A and 16B are process flow diagrams that illustrate interactionsamong the components illustrated in FIG. 15 during a gateway activationprocess, according to at least some embodiments. The activation processinvolves two points of interaction from the customer's perspective.First, the customer interacts with the gateway 84, as shown in FIG. 16A.Second, the customer interacts with the service provider (SP) console68, as shown in FIG. 16B.

FIG. 16A illustrates interactions among the customer (represented bynetwork administrator process 90 in FIG. 15), gateway 84, and theservice provider (SP) gateway control 70 during the activation process.After the gateway 84 is installed and/or powered on, the gateway 84generates a public key (e.g., an RSA keypair), and collects metadataabout the hardware and/or software of the device that the gateway 84 hasbeen installed on. For example, the metadata may include an IP address,a MAC address, or other hardware and software characteristics of thedevice. The gateway 84 then publishes the public key and the metadata,for example via an HTTP POST, to gateway control 70. In response,gateway control 70 may generate an activation key, and returns theactivation key to the gateway 84. The activation key may be a globallyunique identifier (GUID), for example an N-bit, randomly generatednumber. Gateway control 70 may store the activation key along with thepublic key and the metadata obtained from the gateway 84.

After receiving the activation key from gateway control 70, the gateway84 advertises the activation key within the client network 80 at a fixedport (IP address:port) on the gateway 84 VM or device. The customer, vianetwork administrator process 90, may then access the fixed port of thegateway 84 to obtain the activation key; the access is redirected to theservice provider (SP) console 68 process with the activation key in thequery string.

In at least some embodiments, the activation key is valid for a fixedtime or lifespan (for example, 30 minutes), after which the activationkey expires. In at least some embodiments, since the activation key isvalid only for a specified lifespan, a background garbage collectionprocess may be provided at the service provider 60 that removes expiredactivation keys. In at least some embodiments, the lifespan for anactivation key may be longer on the service provider 60 side than on thegateway 84 to handle borderline cases (for example, 45 minutes on theservice provider 60 side, 30 minutes on the gateway 84).

FIG. 16B illustrates interaction among the customer (represented bynetwork administrator process 90 in FIG. 15), service provider (SP)console 68, and the service provider (SP) gateway control 70 during theactivation process. Once the network administrator process 90 hasobtained the activation key from the gateway 84, the activation key maybe used to add the gateway 95 to the customer's service provider 60account. After being redirected to the SP console 68, the customer logsin to the account (e.g., via network administrator process 90), and theactivation key from the query string is used to fetch the metadata thatthe gateway 84 published to the gateway control 70. At least some ofthis metadata is displayed to the customer (e.g., via networkadministrator process 90). The metadata returned from gateway control 70to the SP console 68 and displayed to the customer 90 is the metadatapreviously provided to gateway control 70 by the gateway 84, and may beused to inform the customer 90 about the gateway 84 to be activated. Thedisplayed metadata may confirm to the customer 90 that the respectivegateway 84 indicated by the metadata is the gateway 84 that has beeninstalled at the customer's network. For example, an IP address of thegateway 84 may be displayed, which the customer 90 may confirm is the IPaddress of the gateway 84. In addition, the credentials (e.g, customeraccount number and/or other customer identification information)obtained from the customer 90 to log in to the account may be used inauthenticating the customer 90 as the customer who owns the respectivegateway 84 and associating the customer 90 with the respective gateway84.

The customer 90 may also be prompted, by SP console 68, to enteradditional information, for example a name for the gateway 84. Afterviewing and verifying the displayed metadata, the customer 90 mayauthorize registration of the gateway 84 with gateway control 70 via SPconsole 68, for example by selecting a “confirm” or “activate” or“register” user interface element. When the customer 90 authorizesregistration of the gateway 84 via SP console 68, SP console 68 may passthe activation key obtained from the customer 90 to gateway control 70.Customer information such as a customer-supplied name for the gateway84, the customer account ID, and so on, may also be passed to gatewaycontrol 70. The customer-supplied activation key is matched against theactivation key previously provided to gateway control 70 by gateway 84.The customer information (e.g., the name of the gateway 84) is stored bygateway control 70 along with, for example, the metadata previouslyprovided by the gateway 84.

In at least some embodiments, all data exchanged between SP console 68and SP gateway control 70, and between gateway 84 and SP gateway control70, may be encrypted. In at least some embodiments, sensitive data suchas the customer's credentials, access key or secret key is not passed inthe activation process.

Referring again to FIG. 16A, in at least some embodiments, the SPgateway control 70 is responsible for maintaining all informationpertaining to registration and activation of the gateway 84. The gateway84 meanwhile continuously polls SP gateway control 70 asking forinformation to generate a certificate signing request (CSR). Once SPgateway control 70 has received authorization from the customer 90 viaSP console 68 as illustrated in FIG. 16B and matches thecustomer-supplied activation key to the activation key provided bygateway 84, SP gateway control 70 may respond to the gateway 84 GETrequest by providing metadata including but not limited to at least someof the customer information received from the customer 90 as indicatedin FIG. 16B. The gateway 84 then generates a CSR and sends to SP gatewaycontrol 70. In response to the CSR, SP gateway control 70 generates acertificate and signs the certificate with gateway 84's previouslyprovided public key. In at least some embodiments, the certificate maycontain customer and/or gateway information, for example the customeraccount ID and the customer-supplied gateway 84 name. SP gateway control70 then responds by sending the self-signed certificate, encrypted withthe public key previously provided by gateway 84, to the gateway 84. Thecertificate may then be used for authentication in future communicationsfrom the gateway 84 to the service provider 60.

In at least some embodiments, to help prevent a customer from activatingmultiple gateways 84 using the same activation key,system/hardware-specific information may also be included along with theactivation key which is published to the SP gateway control 70 by thegateway 84.

FIGS. 17A and 17B are a flowchart of the activation process from theperspective of a storage gateway, according to at least someembodiments. As indicated at 500 of FIG. 17A, after the gateway isinstalled and/or powered on, the gateway checks persistent storage todetermine if it has already been activated. For example, the gateway mayhave been powered down for upgrade, maintenance, or for some otherreason. If the gateway has been activated, the activation processproceeds to element 530 of FIG. 17B, where the gateway may obtainconfiguration information from the SP gateway control.

At 500 of FIG. 17A, if the gateway has not been previously activated,the activation process proceeds to element 502 of FIG. 17A, where thegateway checks if it has any persisted customer information forgenerating a certificate signing request (CSR). If the gateway has thepersisted customer information, the process proceeds to element 520 ofFIG. 17B. If the gateway does not have the persisted customerinformation, the process goes to element 504 of FIG. 17A. At 504, thegateway generates a public key (e.g., an RSA keypair). The gateway mayalso collect metadata about the hardware and/or software of the devicethat the gateway has been installed on. For example, the metadata mayinclude an IP address, a MAC address, or other hardware and softwarecharacteristics of the device. The gateway then publishes the public keyand metadata to the SP gateway control, as indicated at 506. At 508, thegateway receives an activation key from the SP gateway control. At 510,the gateway advertises the activation key on a fixed port (IPaddress:port) on the service customer's network.

As indicated at 512 through 516 of FIG. 17A, the gateway may then pollthe SP gateway control for customer information that is required forgenerating a CSR. The customer information may include, but is notlimited to, an account ID of the customer and a customer-specified namefor the gateway. At 512, the gateway may pause, e.g. for a minute or forsome other period, and then check to see if it has received theinformation from the SP gateway control. At 514, if the information hasnot been received, then the gateway checks to see if the activation keyhas expired, as indicated at 516. In at least some embodiments, theactivation key is valid for a fixed time or lifespan (for example, 30minutes), after which the activation key expires. At 516, if theactivation key has not expired, then the activation process returns toelement 512 of FIG. 17A to continue polling the SP gateway control. At516, if the activation key has expired, then the activation processreturns to element 504 of FIG. 17A to obtain a new activation key fromthe SP control plane.

At 514 of FIG. 17A, if the customer information has been received fromthe SP gateway control, then the activation process proceeds to element518 of FIG. 17A, where the gateway stores the customer information topersistent memory. In at least some embodiments, the received customerinformation may be encrypted, and therefore the gateway may decrypt theinformation before storing the information. The process then proceeds toelement 520 of FIG. 17B.

Referring to FIG. 17B, at 520, the gateway may check to see if italready has a certificate. At 520, if the gateway does already have acertificate, the process may proceed to element 530 of FIG. 17B, wherethe gateway may obtain configuration information from the SP gatewaycontrol. At 520, if the gateway does not have a certificate, the processproceeds to element 522. At 522, the gateway generates a CSR and sendsthe CSR to the SP control plane. At 524, the gateway receives a securitycertificate from the SP control plane in response to receiving the CSR;the certificate may serve as security credentials for the gateway. At526, the gateway may disable the advertisement of the activation key(see step 510 of FIG. 17A). At 528, the gateway may save its currentstate to persist information (certificate, customer-specified gatewayname etc.) that has been obtained in the activation process.

At this point, the activation process is complete. At 530, the gatewaymay obtain configuration information from the SP gateway control. In atleast some embodiments, once the customer has been notified that thegateway has been successfully activated, the customer may configure theinstalled and activated gateway via the SP console. The SP console mayprovide a user interface, for example a web interface, to which thecustomer can log on to the customer's account, select the gateway (whichmay be identified by the customer-specified name), and specify aconfiguration for the gateway. In at least some embodiments, the SPconsole passes this configuration on to the SP gateway control, whichthen configures the specified gateway via a connection (e.g., andSSL/TCP connection) initiated by the gateway itself.

Activation Key Security

As indicated at 510 of FIG. 17A, the activation key is made available ata public IP address on the service customer's network, and may be passedunencrypted from the customer to the SP console in the query string.Although the activation key has a limited lifespan and the IP address isonly known to the customer, there is still a short window of time inwhich the activation key is exposed at the IP:Port. While the activationkey by itself is no good without the metadata that is also published bythe gateway to the SP gateway control, the gateway may be vulnerable tosome extent during this short window of time. In at least someembodiments, the customer may utilize security groups or other securitymeasures to help prevent malicious users or processes from obtaining anactivation key and activating someone else's gateway. In addition, sincethe customer is required to log in to the SP console process to activatea gateway, the gateway can be matched with the customer account thatattempts to activate it.

Remote Gateway Management Using Gateway-Initiated Connections

Embodiments of a storage gateway may, for example, serve as anon-premise storage device and as an interface between a servicecustomer's network and a storage service provided by a service provider.In at least some embodiments, an installed storage gateway may beactivated, tracked, configured, and managed remotely via gateway controltechnology implemented at the service provider. FIG. 18 is a high-levelblock diagram that illustrates example gateway control architecture thatmay be employed in at least some embodiments. In at least someembodiments, as illustrated in FIG. 18, gateway control 70 may include agroup of two or more gateway control servers 74 (e.g., gateway controlservers 74A, 74B, 74C, . . . ). The multiple gateway control servers 74may provide load balancing and high availability. During operation, at agiven time, a particular installed and activated storage gateway 84 on aservice customer's network 80 is connected to a particular one of thegateway control servers 74. However, note that the storage gateway 84may be connected to a different gateway control server 74 at some othertime.

A gateway control server 74 that is currently connected to storagegateway 84 may manage the storage gateway 84 by sending requests orcommands to the storage gateway 84 via intermediate network 50. Requestsinitiated from the gateway control server 74 to manage the storagegateway 84 may include, but are not limited to, configuration changerequests and operation requests. However, since the storage gateway 84may be deployed behind a client network 80 firewall, a gateway controlserver 74 may not be able to reach the gateway 84 from outside thefirewall unless an exception rule is created for the gateway 84. Inaddition, in at least some embodiments, the security model for thestorage gateway 84 may dictate that external processes, including butnot limited to service provider processes, are not allowed to initiateconnections to the storage gateway 84.

In at least some embodiments, to enable a gateway control server 74 tosend requests or commands to storage gateway 84 while enforcing thesecurity model that does not allow the service provider to establishconnections to the gateway 84, methods and apparatus for remote gatewaymanagement using gateway-initiated connections are provided. In theremote gateway management method, a gateway initiates a connection tothe service provider by sending a connection request. In at least someembodiments, the connection is established to a particular gatewaycontrol server 74 via a load balancer 72. However, the gateway 84 doesnot send requests messages to the service provider via thegateway-initiated connection. Instead, the service provider (e.g., agateway control server 74) holds the connection pending requests to besent to the gateway 84, while the gateway 84 waits for a response. Uponreceiving a request for the gateway 84, for example from a networkadministrator process 90 or some other process on the client network 80on which the gateway 84 is instantiated, the service provider (e.g., agateway control server 74) sends the request to the gateway 84 via thegateway-initiated connection that the service provider (e.g., a gatewaycontrol server 74) has been holding. The gateway 84 may also send aresponse to a request to the service provider 80 via thegateway-initiated connection.

In at least some embodiments, a gateway control server 74 to which aconnection from gateway 84 is established (e.g., gateway control server74A) may register the connection with registration service 76. If agateway control server 74 receives a request for a gateway 74 to whichit does not hold a connection, the gateway control server 74 may querythe registration service 76 to find out which gateway control server 74holds the connection, and forward the request to the gateway controlserver 74 that holds the connection to the gateway 84. In someembodiments, as an alternative, a gateway control server 74 thatreceives a request for a gateway 74 to which it does not hold aconnection may simply broadcast the request to two or more other gatewaycontrol servers 84.

In at least some embodiments, the service provider 80 may employ a pingprocess to monitor the gateway-initiated connections. In the pingprocess, a gateway control server 84 that maintains a connection to agateway 74 may periodically or aperiodically send a ping message to thegateway 84. The gateway 84 responds to the ping message. Upon detectingthat the gateway 84 has not responded to the ping message(s) for somespecified time-out period, the gateway control server 74 may drop theconnection, and may un-register the connection with the registrationservice 76.

In at least some embodiments, the ping messages may be sent to thegateway(s) 74 at periodic intervals. At least some embodiments mayadjust the ping intervals according to the reliability of theconnections to specific gateways 84 so that ping messages are sent atshorter intervals to a gateway 84 for which the connection has beenunreliable and at longer intervals to a gateway for which the connectionhas been generally reliable. The ping interval may be increased overtime to a given gateway 84 as the connection remains reliable, and maybe decreased to a given gateway 84 for which the connection has beenunreliable.

In at least some embodiments, a gateway 84 may detect if itsgateway-initiated connection has been terminated or dropped. Upondetecting that the connection has terminated, the gateway 84 may sendanother connection request to the service provider 80 to re-establishthe connection. Note that the connection may be re-established to adifferent gateway control server 74 than the one that formerly held theconnection. In at least some embodiments, a gateway 84 may determinethat its gateway-initiated connection has been dropped by monitoring theping messages and determining that a ping message has not been receivedover the connection for a specified time-out period.

Thus, in the remote gateway management method, a gateway 84 establishesa connection to the service provider, anticipating and waiting forrequest(s) from the service provider. The service provider holds theconnection pending requests for the gateway 84. Upon receiving a requestfor the gateway 84, the service provider forwards the request to therespective gateway over the gateway-initiated connection. The serviceprovider and the gateway both monitor and manage the connection so that,if the connection drops for some reason, the drop is detected and thegateway 84 re-establishes the connection.

FIG. 19 is a flowchart of a method for remote gateway management using agateway-initiated connection, according to at least some embodiments. Asindicated at 600, the gateway establishes a connection to a gatewaycontrol server via a connection request. For example, the gateway mayestablish an outbound SSL/TCP connection with the gateway control serverthrough a load balancer, as illustrated in FIG. 18, via a connectionrequest. As indicated at 602 of FIG. 19, once the connection to thegateway is established, the gateway control server holds on to theconnection and keeps the connection alive. As indicated at 604 of FIG.19, the gateway control server receives a request for the gateway. Forexample, a gateway control server 74 may receive a configuration requestor operation request for the gateway 84 from the respective networkadministrator process 90 via a console process 68, as illustrated inFIG. 18. After the gateway control server receives the request for thegateway, the gateway control server forwards the request to the gatewayvia the gateway-initiated connection, as indicated at 606 of FIG. 19.

Referring again to FIG. 18, a service customer may access the serviceprovider console 60 to initiate configuration change requests oroperation requests for an indicated storage gateway 84. For example, anetwork administrator, via network administrator process 90. may send arequest to a gateway 84 via a console process 68. The console process 68may then send the request to a gateway control server 74 behind loadbalancer 72. However, the gateway control server 72 to which the consoleprocess 68 sends the request may not be the gateway control server 72that holds the connection to the respective gateway 84. For example,gateway control server 72B may hold the connection to gateway 84, whilethe request for gateway 84 may be sent to gateway control server 72A.Therefore, a gateway control server 72 that receives the request fromconsole process 68 (e.g., gateway control server 72A) may need toforward the request to the gateway control server that holds theconnection to the gateway 84 (e.g., gateway control server 72B) in orderto deliver the request to the appropriate gateway 84. Thus, at leastsome embodiments may provide a method or methods for a gateway controlserver 72 (e.g., server 72A) to get a request for a particular gateway84 received from the console process 68 to the gateway control server 72(e.g. server 72B) that currently holds a connection to the particulargateway 84 indicated by the request.

In some embodiments, to accomplish this, a gateway control server 72(e.g., server 72A) that receives a request for a gateway 84 to which theserver 72 does not hold a connection may broadcast the request to all ofits peer gateway control servers 72. FIG. 20 is a flowchart of a methodfor a gateway control server to broadcast a gateway request to its peerservers, according to some embodiments. As indicated at 620, when eachgateway control server 72 is instantiated, the server 72 may registerwith a registration service 76. When a gateway control server 72 exits,the server 72 is unregistered from the registration service 76. Theregistration service 76 may, for example, be backed by a databaseservice or a distributed storage service. As indicated at 622, a gatewaycontrol server 72 (e.g., server 72A) may receive a request for a gateway84 to which the server 72 does not hold a connection. To broadcast therequest to its peer gateway control servers 72, the gateway controlserver 72 (e.g., server 72A) may poll the registration service 76 todiscover its peer gateway control servers 72 (e.g., servers 72B and72C), as indicated at 624. The gateway control server 72 (e.g., server72A) may then forward the gateway request to all of the servers 72discovered via the registration service 76, as indicated at 626. Thegateway control server 72 that currently holds the connection to thegateway 84 indicated by the request (e.g., server 72B) may then send therequest to the respective gateway 84.

FIG. 21 is a flowchart of an alternative method for getting a gatewayrequest to the appropriate gateway control server, according to at leastsome embodiments. As indicated at 640, when a gateway control server 72(e.g., server 72B) receives a connection request from a gateway 84, theserver 72 registers the pairing with the gateway 84 in the registrationservice 76. As indicated at 642, a gateway control server 72 (e.g.,server 72A) may receive a request for a gateway 84 to which the server72 does not hold a connection. As indicated at 644, the gateway controlserver 72 (e.g., server 72A) that receives the request for a gateway 84to which the server 72 does not hold a connection may then query theregistration service 72 to find out which gateway control server 72(e.g., server 72B) currently holds a connection with the gateway 84, andmay then forward the request to the gateway control server 72 (e.g.,server 72B) indicated by the registration service 76, as indicated at646. The gateway control server 72 that currently holds the connectionto the gateway 84 indicated by the request (e.g., server 72B) may thensend the request to the respective gateway 84 via the gateway-initiatedconnection.

In at least some embodiments, when a request is delivered to and handledby a gateway 84, a status is returned from the gateway 84 to the gatewaycontrol server 72 that currently holds the connection to the gateway 84(e.g., server 72B), which subsequently returns the status to the gatewaycontrol server 72 from which it previously received the forwardedrequest (e.g., server 72A), which then returns the status to the consoleprocess 68. The console process 68 may then provide an indication ofresults of the request to the customer process (e.g., networkadministrator process 90) that initiated the request. If a request failsto reach the target gateway 84 for some reason, for example if thegateway 84 indicated by the request is unavailable or cannot be found,the console process 68 may provide an indication of failure of therequest to the customer process (e.g., network administrator process 90)that initiated the request. The customer process may retry the request,if necessary or desired.

FIG. 22 is a flowchart of a method for establishing, monitoring andmaintaining gateway-initiated connections, according to at least someembodiments. As indicated at 660, a gateway may be instantiated on aclient network. As indicated at 662, after instantiation, the gatewaysends a connection request to the service provider to establish a secureconnection (e.g., an SSL (Secure Socket Layer)/TCP connection) to theservice provider. In at least some embodiments, a gateway controlprocess at the service provider may hold the connection, and mayregister the connection with a registration service, as indicated at664. Requests for the gateway received by the service provider may thenbe forwarded to the gateway over the gateway-initiated connection.

As indicated at 666, the gateway control process may drop theconnection. For example, in at least some embodiments, the gatewaycontrol process may periodically or aperiodically ping the gateway overthe connection and may, upon detecting that the gateway is notresponding to the ping, drop the connection. If registered with aregistration service, the gateway control process may unregister theconnection.

As indicated at 668, the gateway may detect that the connection has beendropped. For example, in at least some embodiments, the gateway controlprocess may periodically or aperiodically ping the gateway over theconnection. The gateway may detect that the connection has been droppedby determining that pings from the service provider are not beingreceived over the connection.

Note that other methods for detecting dropped connections from eitherthe service provider side or the client network/gateway side may beemployed in some embodiments.

Gateway Proxies

FIG. 18, described above, illustrates a service provider network thatincludes a gateway control 70 implemented as a gateway control planethat includes multiple gateway control servers 74. In at least someembodiments, the service provider network may include a gateway proxyplane that includes multiple gateway proxy nodes and that may be used bythe gateway control plane to communicate with storage gateways. Thegateway proxies may be used to hold and manage the gateway-initiatedconnections for the gateway control servers 74. The gateways 84 initiateconnections to the gateway proxies; the gateway proxies may maintain thecommunications channels to the gateways 84, and may help in ensuring thesecure exchange of messages between the service provider (e.g., thegateway control servers 74) and the gateways, as well as helping toprevent misuse such as multiple copies of the same gateway 84.

Gateway-Proxy Interactions

FIG. 23A is a block diagram that broadly illustrates an architecture fora service provider network that includes a gateway proxy plane,according to at least some embodiments. The gateway proxy plane mayinclude two or more proxy nodes 700, a proxy store 702, a client-sideinterface process (CIP) 720 that is exposed to the external network, anda server-side interface process (SIP) 710 between the proxy nodes 700and gateway control server(s) 74 that is not exposed to the externalnetwork. In some embodiments, the gateway proxies 700 may be implementedon the same physical devices as the gateway control server(s) 74. Inother embodiments, the gateway proxies 700 may be implemented onseparate devices than the gateway control server(s) 74.

A storage gateway 84 that is installed and activated initiates a secureconnection request (e.g., an SSL/TCP connection request) to the gatewayproxy nodes 700 via the CIP 720. The proxy node 700 (in this example,proxy node 700B) that receives the connection request examines thegateway's certificate associated with the connection request to find thegateway identifier and customer account identifier of the gateway 84that initiated this connection. The customer and gateway 84 may beauthenticated using the gateway identifier and customer accountidentifier from the certificate. After authenticating the customer andgateway 84, the proxy node 700 then publishes to the proxy store 702that it is the authoritative proxy 700 to communicate with the connectedgateway 84. The proxies (e.g., proxy 700A and 700B) may query the proxystore 702 to discover other proxies that currently hold connections toparticular gateways.

In at least some embodiments, proxy store 702 may be implemented as adatabase. The database may be either a distributed or a centralizeddatabase. In at least some embodiments, the proxy store 702 may storethe following associations:

(gateway ID, account ID, proxy endpoint)

When a message is to be sent to a gateway 84, a proxy 700 may query theproxy store 702 to find which proxy 702 has a connection to the gateway84. In at least some embodiments, there exists only one entry pergateway 84 in the proxy store 702.

Gateway Control Server-Proxy Interactions

FIG. 23B illustrates a gateway control server messaging a gatewaythrough the gateway proxy plane, according to at least some embodiments.As shown in FIG. 23B, in at least some embodiments, the gateway controlserver 74 may have a message that needs to be sent to a particulargateway 84. The gateway control server 74 sends the message to thegateway proxy nodes 700 via the SIP 710. If the proxy node 700 thatreceives the message holds the connection to the gateway 84, the proxynode 700 forwards the message to the gateway 84 via the connection.However, if the proxy node 700 that receives the message does not holdthe connection to the gateway 84, the proxy node 700 queries the proxystore 702 to determine which proxy node 700 holds the connection to thegateway 84, and forwards the message to the authoritative proxy node 700(in this example, proxy 700B). The authoritative proxy node 700 thenforwards the message to the gateway 84 via the connection.

FIG. 23C illustrates a gateway responding to a gateway control serverrequest through the gateway proxy plane, according to at least someembodiments. In at least some embodiments, a response from gateway 84 togateway control server 74 may follow the reverse path that the requestfrom the gateway control server 74 to the gateway 84 followed as shownin FIG. 23B, starting at the CIP 720 receiving the response from gateway84. The CIP 720 sends the response to the proxy node (proxy 700B) fromwhich it received the request. Note that proxy 700B does not know whichgateway control server 74 the response is for. Proxy 700B completes therequest by sending the response to the proxy node (proxy 700A) fromwhich it received the request. Proxy 700A then sends the response to thegateway control server 74 that initiated the request.

Connection Monitoring and Management

In at least some embodiments, a ping process may be implemented that isused by the proxies in managing the gateway-initiated connections. In atleast some embodiments, a gateway 84 initiates a secure connection, e.g.an SSL/TCP connection, to a gateway proxy 700 via the CIP 720, aspreviously described. The gateway proxy 700 may periodically oraperiodically send a ping message to the gateway 84. Each ping messagemay include a timeout; if the gateway 84 does not receive a ping withinthe time interval, it closes the current connection and re-initiates aconnection via the CIP 720. In at least some embodiments, there is onlyone proxy-gateway mapping in the proxy store 702 at any point in time.If a gateway proxy 700 sends a ping and does not get a response from thegateway 84, it closes its connection to the gateway 84.

In at least some embodiments, on every ping, the gateway proxy 700checks to see if it is the authoritative proxy for a given gateway 84 byquerying the proxy store 702 to determine if another proxy 700 haspublished a connection to the gateway 84. If it is not the authoritativeproxy, the proxy 700 closes the connection to the gateway 84. This mayhandle cases where multiple connections to the proxy nodes 700 have beeninitiated by the same gateway 84, for example if the certificate of agateway 84 has been copied to another gateway and both gateways try toinitiate connections.

FIG. 23D illustrates ping message exchange for a gateway proxy plane,according to at least some embodiments. In at least some embodiments, aping in relation to gateway proxies is an end-to-end ping. A reason forpings is that the TCP “keepalive” functionality has a minimum intervalof 2 hours, while embodiments may need to detect connection timeouts orterminations at shorter time intervals.

In at least some embodiments, a ping follows the path as shown in FIG.23D. A gateway proxy node (in this example, proxy 700B) sends a pingmessage via the SIP 710. The message hits one of the gateway proxy nodes700, in this example proxy 700A. Proxy 700A finds the authoritativeproxy 700 (in this example, proxy 700B) for the gateway 84 by queryingthe proxy store 702, and forwards the pin message to proxy 700B. Proxy700B forwards the message to the gateway 84, and the reply from thegateway 84 follows the same path. In at least some embodiments, onceproxy 700B gets a reply to a ping from the gateway 84, it increases itsping interval to the gateway 84. If a gateway 84 connection breaks, theping interval may be reset to a minimum value. Thus, poor gateway-proxyconnections tend to get pinged more often.

The end-to-end ping method described above, in which the proxy 700initiates the ping message by first sending the ping message to the SIP710, may help to ensure that the gateway proxy nodes 700 are reachablefrom the control plane. If a ping fails, the proxy 700 may assume thatit is not reachable from the control plane (e.g., due to a networkpartition) and close the connection to the gateway 84.

Remote Gateway Management Using Long-Polling Connections

In some embodiments, a long polling technique may be used forgateway-initiated connections. Referring back to FIG. 18, long pollingis a polling technique that emulates an information push from a server(e.g., a gateway control server 74) to a client (e.g., the storagegateway 84). In the long polling technique, a client (e.g., the storagegateway 84) initiates a long-polling connection to the server (e.g., agateway control server 74) and requests information from the server asin a standard client/server poll. However, if the server does not haveany information available for the client, instead of sending an emptyresponse, the server holds the client's request and waits forinformation for the client to become available. Once the informationbecomes available, the server (e.g., a gateway control server 74) mayrespond to the client's long polling request, the response including theinformation to be sent to the client (e.g., the storage gateway 84).

In a gateway-initiated connection method that uses long-polling, thegateway 84 establishes a connection to a gateway control server 74 via along polling request. For example, the gateway 84 may establish anoutbound SSL/TCP connection with the gateway control server 74 through aload balancer 72, as illustrated in FIG. 18, via a long polling request.The gateway control server 74 holds on to the request and keeps theconnection alive. The gateway control server 74 receives a request forthe gateway 84. For example, a gateway control server 74 may receive aconfiguration request or operation request for the gateway 84 from therespective network administrator process 90 via a console process 68, asillustrated in FIG. 18. After the gateway control server 74 receives therequest for the gateway 84, the gateway control server 74 sends aresponse to the gateway's long polling request; the response includesthe request for the gateway 84 (e.g., a configuration request oroperation request). In some embodiments, as an alternative, the gatewaycontrol server 74 may send the received request to the gateway 84 on theestablished connection to the gateway that the gateway control server ismaintaining without responding to the long polling request.

Block Storage I/O Operations on a Storage Gateway

Embodiments of a storage gateway may be implemented as a cached gatewayor a shadowing gateway, as previously described. In an exampleembodiment, a cached gateway may be though of as an on-premiseblock-based appliance that leverages on-premise (local) storage for mostfrequent accessed data and remote storage provided by a storage servicefor essentially infinite total capacity. FIG. 6 is a high-level blockdiagram that broadly illustrates the architecture of and data flow in anexample network environment in which an embodiment of a cached gatewayis implemented. A cached gateway may serve as an interface between aservice customer's local network and a storage service at a serviceprovider's network. In at least some embodiments, a cached gateway mayexpose an iSCSI interface to processes on the customer network, althoughother data interfaces may be exposed in some embodiments. As such, thecached gateway may appear as a data interface target (e.g., an iSCSItarget) operating within the client network, e.g., the cached gatewaymay appear on the client network as a storage array. The cached gatewaymay, for example, expose logical unit numbers (LUNs), e.g., block-basedstorage devices such as hard disks, to processes executing on deviceswithin the client network. The processes in turn may initiate datasessions (e.g., SCSI sessions) with LUNs and send data commands (e.g.,SCSI commands) to the cached gateway.

FIG. 24 illustrates a general architecture for and data I/O operationsof a cached gateway, according to at least some embodiments. In general,in a cached gateway 800, when write data is received from a customerprocess 830, the data is appended to a write log 814; the data is lateruploaded to the remote data store 820 from the write log 814 by anupload process. Metadata for the write data relative to a block, e.g.block location, block type, offset(s) and length, may be added to ametadata store 806. In at least some embodiments, the metadata store 806may be implemented as a database, for example a Berkeley database (BDB).A cached gateway 800 may also locally cache at least some data to alocal cache 812, e.g. frequently and/or recently used data, which mayimprove response to customer read requests as some reads may besatisfied from local cache 812 instead of from the remote data store820. Local cache 812 may also be referred to as a read cache. Themetadata store 806 may also contain location and other information forlocally cached read data in local cache 812. While FIG. 24 shows anembodiment in which one metadata store 806 includes both read cacheentries and write cache entries, in some embodiments the read cacheentries and write cache entries may be maintained in separate metadatastores 806. In at least some embodiments, data read requests fromcustomer processes 830 may be serviced from the write log 814 or localcache 812, if possible; the requested data may be fetched from theremote data store 830 if not. Data from the local cache 812 or theremote data store 830 that is fetched and buffered (e.g., to a blockbuffer 804) to satisfy a read request may be updated with data from thewrite log 814, if updates exist in the write log 814 for the data,before the data is returned to the customer process 830 to satisfy theread request.

In at least some embodiments, both the write log 814 and data cache 812may be implemented in a common, local block-based data store 810. Theblock data store 810 may be implemented in volatile memory, non-volatilememory, or in a combination thereof. The block data store 810 may beimplemented on physical memory within the physical device on whichcached gateway 800 is implemented, on memory external to the physicaldevice on which cached gateway 800 is implemented (e.g., on one or morestorage devices allocated to the gateway 800 by the customer), or on acombination thereof.

Write log data and cached read data may both be stored to the block datastore 810 in a block storage format, for example as 4 MB (four megabyte)blocks. The cached read blocks in the block data store 810 may beconsidered as a read cache, and the write log blocks in the block datastore may be considered as a write buffer. The metadata store 806 maycontain entries for locating both read cache 812 blocks and write log814 blocks in the block data store 810. Blocks may be read from the readcache 812 (or from the write log 814) to satisfy read requests, andblocks may be uploaded from the write log 814 to the remote data store820 via an upload process. In at least some embodiments, when uploadinga write block from the write log 814, the uploaded data may be added tothe read cache 812 as a new read block. The uploaded write log 814blocks may be marked as “free” in the block data store 810, and themetadata store 806 appropriately updated to reflect the changes to theblock data store 810.

In at least some embodiments, a write request may modify or mutate onlya relatively small portion of a block. Thus, in at least someembodiments, when uploading a block from write log 814, only the mutatedportion may be uploaded to remote data store 820, for example using adata deduplication technique as previously mentioned. In addition, thewrite log 814 may include two or more overlapping writes (i.e., writesto the same logical block) stored in different write log 814 blocks.When uploading write data from the write log 814, the two or moreoverlapping writes may be combined for uploading. This combining may beperformed outside the data store, e.g. in a block in block buffer 804;the blocks in write log 814 itself are not mutated.

As mentioned above, in at least some embodiments, when uploading a writeblock from the write log 814, the uploaded data may be added to the readcache 812 as a new read block. For at least some cases, for example whena write block includes numerous mutations and/or when a large portion ofthe write block has been mutated, the write block is simply copied tothe read cache 812 as a new read block, and the metadata store 806 isupdated. However, as mentioned above, a write request may modify ormutate only a relatively small portion of a write log 814 block. Thus,in at least some cases, the respective block may first be fetched fromremote data store 820, and the fetched block updated with themutation(s) from the write log 814, before adding the block to the readcache 812, to ensure that the entire block in read cache 812 isup-to-date. As mentioned, the write log 814 may include two or moreoverlapping writes (i.e., writes to the same logical block) stored indifferent write log 814 blocks, and thus the fetched block may beupdated according to one or more write log 814 blocks. In at least someembodiments, the fetched block may be stored to block buffer 804 forupdating from the write log 804 blocks before being added to the readcache 812.

Generally, new writes are stored to previously freed write log 814blocks in the block data store 810; however, if the block data store 810is detected as being full or nearly full, one or more cached read blocksmay be purged to make room for the write data. Note that read blocks maybe purged from the block data store 810 for other reasons, for exampleto clear space for new read data. Different techniques or policies maybe used to purge read blocks from the block data store 810 in variousembodiments. For example, in some embodiments, a least recently used(LRU) policy may be applied to purge the stalest read blocks from theblock data store 810.

In at least some embodiments, the cached gateway 800 may provide aninterface to two or more volumes 822 on the remote data store 820. In atleast some embodiments, a separate write log 814 and read cache 812 maybe maintained by the cached gateway 800 for each volume 822. In at leastsome embodiments, the separate write logs 814 and read caches 812 fortwo or more volumes 822 may be implemented in the same block data store810. However, in at least some embodiments, the write logs 814 and readcaches 812 for different volumes 822 may be logically or physicallyseparated on the block data store 810. In addition, in at least someembodiments, separate metadata stores 806 may be maintained for theseparate volumes 822.

While FIG. 24 shows read cache 812 and write log 814 as logicallyseparate in block data store 810, in at least some embodiments readblocks and write log blocks for a given volume 822 may be physicallyintermixed in block data store 810. For example, a first physical blockmay be a read block, a second through fifth physical blocks may be writeblocks, the next two physical blocks may be read blocks, and so on.

As mentioned, FIG. 24 illustrates a general architecture for and dataI/O operations of a cached gateway, according to at least someembodiments. However, a storage gateway may also be configured as ashadowing gateway, for example as illustrated in FIG. 7. FIG. 25illustrates a general architecture for and data I/O operations of ashadowing gateway, according to at least some embodiments. A shadowinggateway 801 may include a similar architecture, components, and data I/Ooperations as illustrated and described for cached gateway 800 in FIG.24, except that a shadowing gateway 801 does not include a read cache812 or entries in metadata store 806 for the read cache 812, and theread-related operations described above for a cached gateway are notperformed. Write operations for a shadowing gateway may be similar tothose for a cached gateway, except that writes are not added to a readcache. In addition, read and write requests from customer process(es)830 are forwarded to a local data store 840. Write data from the writerequests, however, are shadowed to remote data store 820. In at leastsome embodiments, the write data are appended to the write log 814 inblock data store 810, and the write data in the write log 814 areperiodically or aperiodically uploaded to the remote data store 820,which maintains a snapshot 824 of the primary data store on local datastore 840.

In at least some embodiments, the write log 814 and write operations forcached gateways, for example as illustrated in FIG. 24, and forshadowing gateways, for example as illustrated in FIG. 25, may beoptimized for write performance. In at least some embodiments, at leastsome I/O operations of a gateway 800 may use block data store 810 as asequential data store. In particular, the write log 814 may be treatedas a sequential data structure, and write operations to the write log814 may be implemented as sequential write operations. In at least someembodiments, the write log 814 may be treated as a one-dimensional databuffer implemented as a linear or circular queue. For cached gateways,data downloaded from remote data store 820 may be stored in read cache812 separately from the write data sent from the customer process(es)830 to the gateway 800, which is stored in write log 814. For bothcached gateways and shadowing gateways, write requests may be receivedfrom the customer process(es) 830 in any order (i.e., the write requestsmay be non-ordered or non-sequential), and write data indicated by thenon-ordered write requests received from the customer process(es) 830may be of arbitrary sizes and may be directed to arbitrary locations oroffsets in the target data store. However, the arbitrary write datareceived from the customer process(es) 830 in non-ordered write requestsis sequentially written and appended to the write log 814. In at leastsome embodiments, the appending may be done at a sub-block level; thatis, two or more instances of write data may be appended within the sameblock in the write log 814. Metadata for the updates to the write log814, e.g., offset and length of the write data in the write log 814blocks as well as offset in the target data store, is stored to themetadata store 806.

FIG. 26 is a flowchart of a method for writing to a write log on a blockdata store, according to at least some embodiments. Implementing thewrite log 814 as a sequential data structure, for example as aone-dimensional queue, may enable the I/O handler 802 to performsequential writes of arbitrary write data received from customerprocess(es) 830 to the block data store 810. As indicated at 850, one ormore write requests may be received from a customer process 830. Thewrite requests may be received in any order (i.e., the write requestsmay be non-ordered), and the write data indicated by the write requestsreceived from the customer process(es) 830 may be of arbitrary sizes andmay be directed to arbitrary locations or offsets in the target datastore. As indicated at 852, sequential writes may be performed tosequentially write the arbitrary write data to the write log 814 onblock data store 810. As indicated at 854, the data in the sequentialwrites to the block data store 810 may be written to contiguouslocations in the block data store 810, for example in contiguouslocations (e.g., sectors) on a disk storage device that implements theblock data store 810. Note that contiguous locations may be, but are notnecessarily, within the same write log block. Using sequential writes toa storage device may reduce or eliminate the need to perform randomsector seeks on the underlying storage device. Performing random sectorseeks negatively impacts I/O operations. For example, disk I/Othroughput may be increased by 10× to 100× by using contiguous writeswhen compared to non-sequential, non-contiguous writes that requirerandom sector seeks. As indicated at 856, the metadata store 806 may beappropriately updated to reflect the writes to the write log 814. In atleast some embodiments, metadata for the writes may be sequentiallyadded to the metadata store 806, which may allow reading of the metadatastore 806 by processes that need to access data in the write log 814more efficient than if the metadata was more randomly added to themetadata store 806.

In at least some embodiments, it may not always be possible to write allwrite log 814 data to contiguous locations in the block data store 810.For example, there may be a read cache 812 block between two write log814 blocks. Thus, at 854, embodiments may attempt to write the write log814 data to contiguous locations as much as possible, but may have toskip some locations (e.g., blocks) if the locations are marked as beingused. The metadata store 806 is appropriately updated so that the writelog 814 data can be located, even if the data are not stored incontiguous blocks.

As described above, logically, the arbitrary write data is appended tothe end of the write log. To implement this, in at least someembodiments, the block buffer 804 is reserved in blocks of the same sizeused in the write log 814 (e.g., 4 MB blocks). An allocated buffer blockis appended to until full. Another buffer block may be allocated forappending new write data; full buffer blocks may be asynchronously andsequentially flushed to the write log 814 on the block data store. Fullblocks in the write log 814 may be asynchronously and sequentiallyuploaded to the remote data store 820 by the upload interface; uploadedblocks from the write log 814 may be marked as “free”.

In cached gateway implementations as illustrated in FIG. 24, to maintaindata consistency, read data may need to be merged with write data beforethe gateway 800 returns the requested data to a customer process 830.FIG. 27 is a flowchart of a method for satisfying a read request,according to at least some embodiments of a cached gateway. As indicatedat 860, a read request is received from a customer process 830. In atleast some embodiments, when a read request is received from a customerprocess 830, the gateway 800 looks up the data range of the read in themetadata store 806 to determine if there is data in the write log 814that overlaps the read range. At 862 of FIG. 27, if overlapping data isfound in the write log 814 that fully covers the read range, the datafrom the write log 814 may be used to directly satisfy the read request,as indicated at 864. Otherwise, at 866 of FIG. 27, if overlapping datais found in the write log 814 that partially covers the read range, theread cache 812 may be checked to see if data is present for the datarange, as indicated at 868. If data is in the read cache 812, then oneor more data block(s) may be fetched from the read cache 812, asindicated at 870. Otherwise, one or more blocks may be fetched fromremote data store 820, as indicated at 872. Note that, in someembodiments, blocks may be fetched from both the read cache and remotedata store 820 to satisfy some read requests. At 874 of FIG. 27, thefetched data blocks may then be updated with mutated data from the writelog 814. At 876 of FIG. 27, the mutated data may be returned to therequesting process 830 to satisfy the read request. In some embodiments,the updated blocks may be added to the read cache 812, as indicated at878 of FIG. 27.

In some embodiments, blocks read from the remote data store 820 tosatisfy a read request may be added to the read cache 812 and updatedfrom the write log 814 prior to sending the blocks to the requestingprocess 830. Alternatively, the blocks may be buffered, for example toblock buffer 804, and updated in the buffer. The updated blocks may thenbe sent from the buffer 804 to the requesting process 830 and added tothe read cache 814 from buffer 804.

In some embodiments, blocks in read cache 812 that are to be used tosatisfy a read request may be updated in place with data from the writelog 814 and then sent from the read cache 812 to the requesting process830 to satisfy the read request. Alternatively, the blocks may be readfrom the read cache 812 and buffered, for example to block buffer 804,and updated in the buffer. The updated blocks may then be sent from thebuffer 804 to the requesting process 830 and added to the read cache 814from buffer 804. The previous versions of the blocks in the read cache812 that were read into the buffer may be marked as free and/oroverwritten by the newly updated blocks.

At 866 of FIG. 27, if no overlapping data is found in the write log 814,the read cache 812 may be checked to see if the read request can besatisfied from the read cache 812, as indicated at 880 of FIG. 27. At880 of FIG. 27, if the read request can be satisfied from the read cache812, then data from the read cache 812 may be returned to the customerprocess 830 to satisfy the read request, as indicated at 882 of FIG. 27.At 880 of FIG. 27, if the read request cannot be satisfied from the readcache 812, one or more data block(s) may be fetched from remote datastore 820, as indicated at 884 of FIG. 27. Data from the fetched blocksmay be returned to the customer process 830 to satisfy the read request,as indicated at 886 of FIG. 27. In some embodiments, the blocks fetchedfrom remote data store 820 to satisfy a read request may be added to theread cache 812, as indicated at 888 of FIG. 27.

In at least some embodiments, a gateway 800 may allow customers torequest, a snapshot of the write log 814 to be taken and uploaded to theremote data store 820, for example through a console process provided bythe service provider. In addition, or instead, the gateway 800 mayperiodically or aperiodically automatically take and upload a snapshotof the write log 814 to the remote data store 820. Uploading a snapshotof the write log 814 may, for example, provide protection of data fromhardware and software failures. In at least some embodiments, thesnapshot is a point-in-time snapshot; only mutated data that is in thewrite log at the time the snapshot is requested is uploaded in thesnapshot. In at least some embodiments, for cached gatewayimplementations, when the mutated data is uploaded, the locally storedread cache 812 may also be updated with at least some of the data beinguploaded so that the data does not need to be downloaded from the remotedata store 820 for future reads. After the mutated data is uploaded tothe remote data store 820, the data in the write log 814 and thecorresponding data in the metadata store 806 can be discarded (e.g.,marked as “free”), and the space can be reused.

Coalescing Write Data for Upload to the Remote Data Store

As previously described, write log blocks may be periodically oraperiodically uploaded to the remote data store. In at least someembodiments, a data deduplication technique may be used in uploading thewrite log blocks. However, the described data deduplication techniqueoperates during the upload process on whatever data is in the block(s)that are staged to be uploaded. Since arbitrary writes from the customerprocess(es) are sequentially appended to the write log, and the customerprocess(es) may write more than once to the same location in the targetdata store, a write log block or blocks may include more than one writedirected to the same location (e.g., offset and/or range) of the targetdata store.

Thus, at least some embodiments may implement a pre-upload coalescingtechnique for the write data in the write log blocks. In this technique,the metadata for a write log block (or blocks) being staged foruploading may be examined to determine if there is more than one writein the write log block(s) directed to the same location in the targetdata store. If there is more than one write to given location, then theearlier write(s) may be suppressed when building a buffer block to beuploaded. Thus, a block that is passed to the upload process foruploading, e.g. according to the data deduplication technique, mayinclude only one write (the most recent write) to a given location,rather than possibly two or more writes to the same location that may bepresent if the pre-upload coalescing technique was not applied.

Illustrative System

In at least some embodiments, a computer system that implements aportion or all of one or more of the storage gateway technologies asdescribed herein, may include a general-purpose computer system thatincludes or is configured to access one or more computer-accessiblemedia, such as computer system 3000 illustrated in FIG. 28. In theillustrated embodiment, computer system 3000 includes one or moreprocessors 3010 coupled to a system memory 3020 via an input/output(I/O) interface 3030. Computer system 3000 further includes a networkinterface 3040 coupled to I/O interface 3030.

In various embodiments, computer system 3000 may be a uniprocessorsystem including one processor 3010, or a multiprocessor systemincluding several processors 3010 (e.g., two, four, eight, or anothersuitable number). Processors 3010 may be any suitable processors capableof executing instructions. For example, in various embodiments,processors 3010 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 3010 may commonly,but not necessarily, implement the same ISA.

System memory 3020 may be configured to store instructions and dataaccessible by processor(s) 3010. In various embodiments, system memory3020 may be implemented using any suitable memory technology, such asstatic random access memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of memory. In theillustrated embodiment, program instructions and data implementing oneor more desired functions, such as those methods, techniques, and datadescribed above for storage gateway technologies, are shown storedwithin system memory 3020 as code 3025 and data 3026.

In one embodiment, I/O interface 3030 may be configured to coordinateI/O traffic between processor 3010, system memory 3020, and anyperipheral devices in the device, including network interface 3040 orother peripheral interfaces. In some embodiments, I/O interface 3030 mayperform any necessary protocol, timing or other data transformations toconvert data signals from one component (e.g., system memory 3020) intoa format suitable for use by another component (e.g., processor 3010).In some embodiments, I/O interface 3030 may include support for devicesattached through various types of peripheral buses, such as a variant ofthe Peripheral Component Interconnect (PCI) bus standard or theUniversal Serial Bus (USB) standard, for example. In some embodiments,the function of I/O interface 3030 may be split into two or moreseparate components, such as a north bridge and a south bridge, forexample. Also, in some embodiments some or all of the functionality ofI/O interface 3030, such as an interface to system memory 3020, may beincorporated directly into processor 3010.

Network interface 3040 may be configured to allow data to be exchangedbetween computer system 3000 and other devices 3060 attached to anetwork or networks 3050, such as other computer systems or devices asillustrated in the other Figures described herein, for example. Invarious embodiments, network interface 3040 may support communicationvia any suitable wired or wireless general data networks, such as typesof Ethernet network, for example. Additionally, network interface 3040may support communication via telecommunications/telephony networks suchas analog voice networks or digital fiber communications networks, viastorage area networks such as Fibre Channel SANs, or via any othersuitable type of network and/or protocol.

In some embodiments, system memory 3020 may be one embodiment of acomputer-accessible medium configured to store program instructions anddata as described above in reference to the other Figures forimplementing embodiments of storage gateway technologies. However, inother embodiments, program instructions and/or data may be received,sent or stored upon different types of computer-accessible media.Generally speaking, a computer-accessible medium may includenon-transitory storage media or memory media such as magnetic or opticalmedia, e.g., disk or DVD/CD coupled to computer system 3000 via I/Ointerface 3030. A non-transitory computer-accessible storage medium mayalso include any volatile or non-volatile media such as RAM (e.g. SDRAM,DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc, that may be included in someembodiments of computer system 3000 as system memory 3020 or anothertype of memory. Further, a computer-accessible medium may includetransmission media or signals such as electrical, electromagnetic, ordigital signals, conveyed via a communication medium such as a networkand/or a wireless link, such as may be implemented via network interface3040.

CONCLUSION

Various embodiments may further include receiving, sending or storinginstructions and/or data implemented in accordance with the foregoingdescription upon a computer-accessible medium. Generally speaking, acomputer-accessible medium may include storage media or memory mediasuch as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile ornon-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.),ROM, etc, as well as transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as network and/or a wireless link.

The various methods as illustrated in the Figures and described hereinrepresent exemplary embodiments of methods. The methods may beimplemented in software, hardware, or a combination thereof. The orderof method may be changed, and various elements may be added, reordered,combined, omitted, modified, etc.

Various modifications and changes may be made as would be obvious to aperson skilled in the art having the benefit of this disclosure. It isintended to embrace all such modifications and changes and, accordingly,the above description to be regarded in an illustrative rather than arestrictive sense.

1. A method, comprising: implementing, at a service provider, a plurality of gateway control servers; receiving, at a particular one of the plurality of gateway control servers, a connection request from a particular storage gateway on a customer network, wherein the storage gateway operates as an interface between one or more customer processes on the customer network and the service provider to store customer data to a remote data store provided by the service provider; in response to the connection request, holding, by the particular gateway control server, a connection to the particular storage gateway, wherein the storage gateway waits to receive requests from the service provider via the connection and does not initiate requests to the gateway control server via the connection; obtaining, by the service provider, information for the particular storage gateway; providing the information for the particular storage gateway to the particular gateway control server; and sending, by the particular gateway control server, one or more request messages to the particular storage gateway via the connection, wherein said one or more request messages include the obtained information.
 2. The method as recited in claim 1, further comprising the particular gateway control server continuing to hold the connection to the particular storage gateway after said sending the one or more request messages to the particular storage gateway.
 3. The method as recited in claim 1, wherein said providing the information for the particular storage gateway to the particular gateway control server comprises: receiving, by a different gateway control server, the information for the particular storage gateway, wherein the different gateway control server does not hold a connection to the particular storage gateway; discovering, by the different gateway control server, the particular gateway control server that holds the connection to the particular storage gateway; and forwarding the information for the particular storage gateway to the particular gateway control server that holds the connection to the particular storage gateway.
 4. The method as recited in claim 1, wherein said providing the information for the particular storage gateway to the particular gateway control server comprises: receiving, by a different gateway control server, the information for the particular storage gateway, wherein the different gateway control server does not hold a connection to the particular storage gateway; and broadcasting, by the different gateway control server, the information for the particular storage gateway to the plurality of gateway control servers.
 5. The method as recited in claim 1, further comprising the particular gateway control server dropping the connection to the particular storage gateway upon detecting that the particular storage gateway is no longer connected.
 6. The method as recited in claim 5, wherein said detecting that the particular storage gateway is no longer connected comprises: periodically or aperiodically sending ping messages to the particular storage gateway over the connection, wherein the storage gateway is operable to respond to received ping messages; and detecting that the particular storage gateway is no longer connected by determining that the particular storage gateway is not responding to the ping messages over the connection.
 7. A device, comprising: at least one processor; and a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to implement a gateway control process operable to: receive a connection request from a storage gateway on a customer network, wherein the storage gateway operates as an interface between one or more customer processes on the customer network and a service provider to store customer data to a remote data store provided by the service provider; in response to the connection request, hold a connection to the storage gateway, wherein the storage gateway waits to receive requests from the service provider via the connection and does not initiate requests to the gateway control server via the connection; obtain information for the storage gateway; and send a request message to the storage gateway via the connection, wherein said request message includes the obtained information.
 8. The device as recited in claim 7, wherein the gateway control process is further operable to continue to hold the connection to the storage gateway after said sending the request message to the storage gateway.
 9. The device as recited in claim 7, wherein, to obtain information for the storage gateway, the gateway control process is further operable to receive the information from a second gateway control process that does not hold a connection to the storage gateway.
 10. The device as recited in claim 9, wherein the information is received from the second gateway control process as a forwarded message specifically to the gateway control process that holds the connection to the storage gateway.
 11. The device as recited in claim 10, wherein the gateway control process registers the connection to the storage gateway with a registration service, and wherein the second gateway control process discovers the gateway control process that holds the connection to the respective storage gateway via the registration service.
 12. The device as recited in claim 9, wherein the information is received from the second gateway control process as a broadcast message from the second gateway control process to one or more gateway control processes.
 13. The device as recited in claim 12, wherein each gateway control process registers its presence with a registration service, and wherein the second gateway control process discovers the one or more gateway control processes via the registration service.
 14. The device as recited in claim 7, wherein, to obtain information for the storage gateway, the gateway control process is further operable to receive the information from a console process of the service provider, wherein the console process received the information from an administrative process on the customer network.
 15. The device as recited in claim 7, wherein the information includes configuration or management instructions for the storage gateway.
 16. The device as recited in claim 7, wherein the gateway control process is one of a plurality of gateway control processes, and wherein only one connection from the storage gateway is held by the plurality of gateway control processes at a given time.
 17. The system as recited in claim 7, wherein the gateway control process is further operable to drop the connection to the storage gateway upon detecting that the storage gateway is no longer connected.
 18. The system as recited in claim 17, wherein, to detect that the storage gateway is no longer connected, the gateway control process is further operable to: periodically or aperiodically send ping messages to the storage gateway over the connection, wherein the storage gateway is operable to respond to received ping messages; and determine that the storage gateway is not responding to the ping messages over the connection.
 19. A non-transitory computer-accessible storage medium storing program instructions computer-executable to implement a gateway control process operable to: receive a connection request from a gateway process on a customer network, wherein the gateway process operates as an interface between one or more customer processes on the customer network and a service provided by a remote service provider; in response to the connection request, hold a connection to the gateway process, wherein the storage gateway waits to receive requests from the service provider via the connection and does not initiate requests to the gateway control server via the connection; obtain information for the gateway process; and send one or more request messages to the gateway process via the long polling connection, wherein said one or more request messages include the obtained information.
 20. The non-transitory computer-accessible storage medium as recited in claim 19, wherein the gateway control process is further operable to: obtain subsequent information for the gateway process; and send one or more subsequent responses to the gateway process via the connection, wherein said additional responses include the additional information. 21.-35. (canceled) 